How to publish builds to Artifactory from GitLab CI? - artifactory

I am looking for an easy and clean way to publish artefacts build with GitLab CI onto Artifactory.
I was able to spot https://gitlab.com/gitlab-org/omnibus/blob/af8af9552966348a15dc1bf488efb29a8ca27111/lib/omnibus/publishers/artifactory_publisher.rb but I wasnt able to find any documentation regarding how I am supposed to configure it to make it work.
Note: I am looking for a gitlab_ci.yaml approach, not as in implementing it externally.

At a basic level, this can be done with the JFrog CLI tools. Unless you want to embed configuration in your .gitlab-ci.yml (I don't) you will first need to run (on your runner):
jfrog rt c
This will prompt for your Artifactory URL and an API key by default. After entering these items, you'll find ~/.jfrog/jfrog-cli.conf containing JSON like so:
{
"artifactory": {
"url": "http://artifactory.localdomain:8081/artifactory/",
"apiKey": "AKCp2V77EgrbwK8NB8z3LdvCkeBPq2axeF3MeVK1GFYhbeN5cfaWf8xJXLKkuqTCs5obpzxzu"
}
}
You can copy this file to the GitLab runner's home directory - in my case, /home/gitlab-runner/.jfrog/jfrog-cli.conf
Once that is done, the runner will authenticate with Artifactory using that configuration. There are a bunch of other possibilities for authentication if you don't want to use API keys - check the JFrog CLI docs.
Before moving on, make sure the 'jfrog' executable is in a known location, with execute permissions for the gitlab-runner user. From here you can call the utility within your .gitlab-ci.yml - here is a minimal example for a node.js app that will pass the Git tag as the artifact version:
stages:
- build-package
build-package:
stage: build-package
script:
- npm install
- tar -czf test-project.tar.gz *
- /usr/local/bin/jfrog rt u --build-name="Test Project" --build-number="${CI_BUILD_TAG}" test-project.tar.gz test-repo

If you're building with maven this is how I managed to do mine:
Note: you need to have your artifactory credentials (user and pass) ready.
Create a master password and generate an encrypted password from it. The procedure on how to create a masterpassword can be found here
In your pipeline settings in gitlab, create 2 secret variables, one for the username and the other for your encrypted password.
Update or create a settings.xml file in .m2 directory for maven builds. Your settings.xml should look like this:
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<servers>
<server>
<id>central</id>
<username>${env.ARTIFACTORY_USER}</username>
<password>${env.ENCRYPTED_PASS}</password>
</server>
</servers>
</settings>
In your .gitlab-ci.yml file, you need to use this settings.xml like this:
image: maven:latest
variables:
MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
cache:
paths:
- .m2/repository/
- target/
build:
stage: build
script:
- mvn $MAVEN_CLI_OPTS compile
and that's it. This should work. You can visit here for more about how to use artifactory with maven

I know this doesn't exactly answer your question, but I got to this question from a related search, so I thought it might be relevant to others too:
I ended up using an mvn deploy job that was bound to the deploy stage for gitlab.
Here is the relevant job portion:
deploy:jdk8:
stage: test
script:
- 'mvn $MAVEN_CLI_OPTS deploy site site:stage'
only:
- master
# Archive up the built documentation site.
artifacts:
paths:
- target/staging
image: maven:3.3.9-jdk-8

Related

Cannot find file './aws-exports' in './src'

I'm on the third module of this AWS tutorial to build a React app with AWS, Amplify and GraphQL but the build keeps breaking. When I ran amplify push --y the CLI generated ./src/aws-exports.js and added the same file to the .gitignore. So I'm not surprised the build is failing, since that file isn't included when I push my changes.
So I'm not sure what to do here. Considering it's automatically added to the .gitignore I'm hesitant to remove it.
Any suggestions?
I'm assuming you are trying to build your app in a CI/CD environment?
If that's the case then you need to build the backend part of your amplify app before you can build the frontend component.
For example, my app is building from the AWS amplify console and in my build settings I have
version: 0.1
backend:
phases:
build:
commands:
- '# Execute Amplify CLI with the helper script'
- amplifyPush --simple
frontend:
phases:
preBuild:
commands:
- yarn install --frozen-lockfile
build:
commands:
- yarn build
artifacts:
baseDirectory: build
files:
- "**/*"
cache:
paths:
- node_modules/**/*
Note that the backend is building first with the amplifyPush --simple command. This is what generates the aws-exports.js file.
The 'aws-exports.js' file gets created automatically when AWS Amplify runs the CI/CD deployment build process and gets configured with the appropriate settings for the environment you are deploying to.
And for this reason it is included in the .gitignore. You don't want your local test configuration to be used in your production deployment for example.
As per Matthe's answer above the should be generated when the build script runs the 'amplifyPush' command. For some reason this is not working for me at the moment though!
AWS added support to automatically generate the aws-exports.js at build time to avoid getting the error: https://docs.aws.amazon.com/amplify/latest/userguide/amplify-config-autogeneration.html

How to utilize .ebextension while using CodePipeline

I'm using CodePipeline to deploy whatever is on master branch of the git to Elastic Beanstalk.
I followed this tutorial to extend the default nginx configuration (specifically the max-body-size): https://medium.com/swlh/using-ebextensions-to-extend-nginx-default-configuration-in-aws-elastic-beanstalk-189b844ab6ad
However, because I'm not using the standard eb deploy command, I dont think the CodePipeline flow is going into the .ebextension directory and doing the things its supposed to do.
Is there a way to use code pipeline (so i can have CI/CD from master) as well as utilize the benefits of .ebextension?
Does this work if you use the eb deploy command directly? If yes, then I would try using the pipeline execution history to find a recent artifact to download and test with the eb deploy command.
If CodePipeline's Elastic Beanstalk Job Worker does not play well with ebextensions, I would consider it completely useless to deploy to Elastic Beanstalk.
I believe there is some problem with the ebextensions themselves. You can investigate the execution in these log files to see if something is going wrong during deployment:
/var/log/eb-activity.log
/var/log/eb-commandprocessor.log
/var/log/eb-version-deployment.log
All the config files under .ebextension will be executed based on the order of precedence while deploying on the Elastic Beanstalk. So, it is doesn't matter whether you are using codepipeline or eb deploy, all the file in ebextension directory will be executed. So, you don't have to worry about that.
Be careful about the platform you're using, since “64bit Amazon Linux 2 v5.0.2" instead of .ebextension you have to use .platform.
Create .platform directory instead of .ebextension
Create the subfolders and the proxy.conf file like in this path .platform/nginx/conf.d/proxy.conf
In proxy.conf write what you need, in case of req body size just client_max_body_size 20M;
I resolved the problem. You need include .ebextension folder in your deploy.
I only copy the dist files, then I need include too:
- .ebextensions/**/*
Example:
## Required mapping. Represents the buildspec version. We recommend that you use 0.2.
version: 0.2
phases:
## install: install dependencies you may need for your build
install:
runtime-versions:
nodejs: 12
commands:
- echo Installing Nest...
- npm install -g #nestjs/cli
## pre_build: final commands to execute before build
pre_build:
commands:
- echo Installing source NPM dependencies...
- npm install
## build: actual build commands
build:
commands:
# Build your app
- echo Build started on `date`
- echo Compiling the Node.js code
- npm run build
## Clean up node_modules to keep only production dependencies
# - npm prune --production
## post_build: finishing touches
post_build:
commands:
- echo Build completed on `date`
# Include only the files required for your application to run.
artifacts:
files:
- dist/**/*
- package.json
- node_modules/**/*
- .ebextensions/**/*
Ande the config file /.ebextensions/.node-settings.config:
option_settings:
aws:elasticbeanstalk:container:nodejs:
NodeCommand: "npm run start:prod"

Bitbucket pipelines: The type or namespace name could not be found

I'm hoping someone can help me out. Thanks!
I'm trying to deploy my dotnet core 2.0 API
When I try and build the project with bitbucket pipelines I get multiple errors finding references. It does restore the project successfully.
However the project builds successfully on my laptop.
folder structure:
/API
/Controllers
/Migrations
/Models
/Services
API.csproj
Program.cs
Startup.cs
bitbucket-pipelines.yml
pipelines:
default:
- step:
image: microsoft/dotnet
name: Check if it builds
script:
- cd API
- dotnet build
example error:
Services/MyService.cs(18,29): error CS0246: The type or namespace name 'IRepository<>' could not be found (are you missing a using directive or an assembly reference?) [/opt/atlassian/pipelines/agent/build/API/API.csproj]
Note I have the latest version of dotnet, same as I'm using in bitbucket pipelines. I have checked via running dotnet --info
Finally i've figured out what was the cause of this issue. I feel really silly for not figuring this out sooner.
My git repository was somehow setup with ignorecase = true.
I have switched it to false (which will prevent this issue in the future).
This means that I can have two of the same files or folders.
I had renamed a folder to a different case.
My repo had
API/
API.csproj
and
api/
api.csproj
My Mac couldn't allow for both so I only saw one folder and one project on my local machine.
To fix this, I had to git rm -r --cached api
This deleted the duplicate folder
I had the project file as a duplicate as well so used git rm -f api.csproj to remove the file from the repository.
Then git pull to bring those changes into my local master branch.

Unable to publish to an NPM Registry (local)

I am running Nexus 3.0.1-01, and am using it to host both Maven repositories and NPM registries. For NPM, I have a local mirror of npmjs.org, a local NPM registry and a group that combines the two...
I have been using this with npm internally, where I can use the npm-public group as my registry and this has been working fine. So, I can use Nexus to mirror npmjs.
The next step is to take locally written npm modules and publish them to npm-releases (on my Nexus instance) so that these modules can be shared amongst the delivery teams here. I've been able to build out a package, and npm pack seems to behave.
I have run npm adduser to provide my Nexus credentials to my npm environment. I am using the same username/password I use when I log into the Nexus web app, and my user is assigned the admin role (so I should have all permissions). I can see the credentials in my .npmrc file
My registry value is still the npm-public group which combined the mirror and my local registry. I have ensured that the package.json of the module I am attempting to deploy has a "publishConfig" section that points to the url of the local registry (not the public group)
However, despite all of that, calling "npm publish" results in a 401 error...
Looking at the npm-debug.log, I can see it's attempting to call the HTTP PUT call to push the assembled tgz file to the registry, and this is returning a 401 error
I have enabled debug logging on the shiro package in the server, but I only ever see a single message thinking it needs authentication
2016-09-13 08:56:28,590+1000 DEBUG [qtp1257823896-4030] *UNKNOWN org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter - Authentication required: sending 401 Authentication challenge response.
According to the documentation, I've done all I'm supposed to do (npm add user) but I am unable to successfully deploy an npm module to my local repository.
Are there additional options I need to use when calling npm publish? Are there additional settings I need to make to the hosted npm repository in my server that will allow me to publish to it? I pretty much created it using the default values. Have I missed a step that is preventing me from deploying to my hosted npm registry?
Any help would be greatly appreciated, as I am stuck at the moment.
As it turns out, I did not have the Npm Bearer Token Realm in my list of active realms. Once I moved it to Active, the publish completed successfully!
Additonally to #EdH's answer we discovered that the format of .npmrc has changed so base64 encoded _auth will not work anymore... and the token has to be created by logging into the repo.
old .npmrc
registry=https://host/repo
_auth=12afdjsljl123213
new .npmrc
//host/repo/:_authtoken=uuidOfToken
Additionally to #Daniel's answer, I discovered that when adding user credentials/logging into npm, you can't have a trailing slash on the registry url.
Bad:
npm adduser --registry=https://repo.localhost/repository/npm-internal/
Good:
npm adduser --registry=https://repo/repository/npm-internal
Also, if you're looking to automate the login (i.e. non-interactively), I used a good tool called npm-cli-login.
npm-cli-login -u admin -p admin123 -e nick#foo.bar -r https://repo/repository/npm-internal
I realize that this post is a couple years old now, but as of struggling with this issue for many hours, I finally found what worked for me that I've not seen in many other places. The problem was that I was not specifying the _authToken within the .npmrc file. Adding this as shown below resolved the issue and I was able to successfully run npm commands without a 401 error.
.npmrc
registry=https://test.repo.com/repository/npm-group/
//test.repo.com/repository/npm-group/:_authToken=NpmToken.${NPM_TOKEN}
Another reason why you might get such an error is that you might have enabled anonymous access to the server for the 'npm Bearer Token Realm'. After disabling anonymous access you must also remove .npmrc and re-add the registry and adduser.

Deployment with Password using Linux in Meteor?

How to deploy a app with password in meteor using linux. It's deploy good when with out set password.But i need to deploy app with password in meteor. I did but gets some error messages i didn't understand the following messages.So please see the below deployment process and suggestions me what to do?
[root#localhost myapp]# meteor deploy myapp.meteor.com --password
deploy: the --password option needs a value.
Try 'meteor help deploy' for help.
[root#localhost myapp]# meteor deploy myapp.meteor.com 123456
deploy: too many arguments.
Usage: meteor deploy <site> [--settings settings.json] [--debug] [--delete]
Deploys the project in your current directory to Meteor's servers.
You can deploy to any available name under 'meteor.com'
without any additional configuration, for example,
'myapp.meteor.com'. If you deploy to a custom domain, such as
'myapp.mydomain.com', then you'll also need to configure your domain's
DNS records. See the Meteor docs for details.
The --settings flag can be used to pass deploy-specific information to
the application. It will be available at runtime in Meteor.settings, but only
on the server. If the object contains a key named 'public', then
Meteor.settings.public will also be available on the client. The argument
is the name of a file containing the JSON data to use. The settings will
persist across deployments until you again specify a settings file. To
unset Meteor.settings, pass an empty settings file.
The --delete flag permanently removes a deployed application, including
all of its stored data.
Options:
--delete, -D permanently delete this deployment
--debug deploy in debug mode (don't minify, etc)
--settings set optional data for Meteor.settings
--star a star (tarball) to deploy instead of the current Meteor app
[root#localhost myapp]#
Perhaps you should use both command and option:
meteor deploy myapp.meteor.com --password 123456

Resources