I'm trying to use Concourse to grab a dockerfile defintion from a git repository, do some work, build the docker image, and push the new image to Artifactory. See below for the pipeline definition. At this time I have all stages up to the artifactory stage (the one that pushes to Artifactory) working. The artifactory stage exits with error with the following output:
waiting for docker to come up...
sha256:c6039bfb6ac572503c8d97f42b6a419b94139f37876ad331d03cb7c3e8811ff2
The push refers to repository [artifactory.server.com:2077/base/golang/alpine]
a4ab5bf94afd: Preparing
unauthorized: The client does not have permission to push to the repository.
This would seem straight-forward as an Artifactory permissions issue, except that I've tested locally with the docker cli and am able to push using the same user/pass as specified within destination_username and destination_password. I double checked the credentials to make sure I'm using the same ones and find that I am.
Question #1: is there any other known cause for getting this error? I've scoured the resource github page without finding anything. Any ideas why I may be getting the permissions error?
Without having an answer to the above question, I'd really like to dig deeper into troubleshooting the problem. To do so I use fly hijack to get a shell in the corresponding container. I notice that docker is installed on the container, so next step I think would be to do a docker import on the tarball for the image I'm trying to push and then perform a docker push to push it to the repo. When attempting to run the import I get the error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is
the docker daemon running?
Question #2: Why can't I use docker commands from within the container? Perhaps this has something to do with the issue I'm seeing with pushing to repo when running the pipeline (I don't think so)? Is it because the container isn't running with privilege? I thought that the privileged argument would be supplied in the resource type definition, but if not, how can I run with privilege?
resources:
- name: image-repo
type: git
source:
branch: master
private_key: ((private_key))
uri: ssh://git#git-server/repo.git
- name: artifactory
type: docker-image
source:
repository: artifactory.server.com:2077/((repo))
tag: latest
username: ((destination_username))
password: ((destination_password))
jobs:
- name: update-image
plan:
- get: image-repo
- task: do-stuff
file: image-repo/scripts/do-stuff.yml
vars:
repository-directory: ((repo))
- task: build-image
privileged: true
file: image-repo/scripts/build-image.yml
- put: artifactory
params:
import_file: image/image.tar
Arghhhh. Found after much troubleshooting that the destination_password wasn't being picked up properly due to special characters and a lack of quotes. Fixed the issue by properly setting the password within yaml file being included with the --load-vars flag.
Related
The Jelastic Marketplace is full of interesting software. However, sometimes, they do not comply to my security needs. In those cases, I would like to write my own manifest that would install the manifest from the marketplace and add up the components that I need for my use-case. Let's take an example: I would like to wrap the kubernetes installation with the addition of a load-balancer. I would like to do something like this:
type: install
name: My Example Manifest
onInstall:
- install:
jps: https://github.com/jelastic-jps/kubernetes/blob/1.23.6/manifest.jps
envName: env-${fn.random}
settings:
deploy: cmd
cmd: echo "do nothing"
topo: 0-dev
dashboard: general
ingress-controller: Nginx
storage: true
api: true
monitoring: true
version: 1.23.6
jaeger: false
- addNodes:
- nodeType: nginx-dockerized
nodeGroup: bl
count: 1
fixedCloudlets: 1
flexibleCloudlets: 4
The issue I am having here is that the manifest cannot add the nodes, because of the following error:
user [xyz] doesn't have any access rights to app [dashboard]
What am I doing wrong? How can I make this manifest work? I tried to set user: root in the addNodes function but it doesn't help.
Of course, I am interested in suggestions involving one single install manifest. I know I could make it happen by first installing the kubernetes manifest and then running an update manifest that would add my load-balancer nodes. I would like, however, to package the whole thing within one single step, as described by my manifest above.
Here's a transcript so far:
$ sbt new lagom/lagom-scala.g8
[info] Loading global plugins from /Users/abrahma/.sbt/1.0/plugins
[info] Set current project to lagomlife (in build file:/Users/abrahma/Bitbucket/Practice-Scala/LagomLife/)
[info] Set current project to lagomlife (in build file:/Users/abrahma/Bitbucket/Practice-Scala/LagomLife/)
ssh://git#github.com/lagom/lagom-scala.g8.git: Auth fail
I've verified authentication with
GitHub:
$ ssh -T git#github.com
Hi agam! You've successfully authenticated, but GitHub does not provide shell access.
Also verified that I can access the repo in question (i.e. I can do the following, in a separate location):
git clone ssh://git#github.com/lagom/lagom-scala.g8.git
Edit: fwiw I'm able to work around whatever the root cause is here:
git clone ssh://git#github.com/lagom/lagom-scala.g8.git
g8 file:///Users/abrahma/tmp/lagom-scala.g8
I ran into this problem as well, and solved it by removing the following from my ~/.gitconfig:
[url "git#github.com:"]
insteadOf = https://github.com/
Are you using OpenSSH 7.8 or newer, and have you recently created your private key?
If so you might be running into the issue described in “Invalid privatekey” when using JSch aka jsch#129.
The root cause was discovered to be the ssh private key mismatch.
The question has a workaround to convert the key file to an older format if that's the error you're seeing.
You need to add your SSH key to the agent:
ssh-add ~/.ssh/id_rsa
is it possible to proxy a private docker registry that runs on docker distribution using nexus oss 3?
i am able to successfully proxy the docker hub, however when i try to proxy my own internal registry, i just end up with image not found errors.
2016-08-31 15:58:21,457+0000 WARN [qtp331814152-140] admin org.sonatype.nexus.repository.docker.internal.V1Handlers - Error: GET /v1/repositories/company-npm/images: 404 - org.sonatype.nexus.repository.docker.internal.V1Exception$ImagesNotFound: images not found
2016-08-31 15:58:30,764+0000 WARN [qtp331814152-140] admin org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: GET /v2/library/company-java/manifests/latest: 404 - org.sonatype.nexus.repository.docker.internal.V2Exception: manifest unknown
2016-08-31 15:58:30,811+0000 WARN [qtp331814152-51] admin org.sonatype.nexus.repository.docker.internal.V1Handlers - Error: GET /v1/repositories/company-java/images: 404 - org.sonatype.nexus.repository.docker.internal.V1Exception$ImagesNotFound: images not found
2016-08-31 15:58:46,379+0000 WARN [qtp331814152-164] admin org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: GET /v2/library/company-java/manifests/6.0.0: 404 - org.sonatype.nexus.repository.docker.internal.V2Exception: manifest unknown
the documentation for the feature does not seem to indicate if this is supported.
I had this same issue with Nexus 3.0.1-01. For me the problem came down to namespacing. Nexus inserts the /library namespace for all repository access commands when a namespace is left blank. See https://books.sonatype.com/nexus-book/3.0/reference/docker.html section 9.8.
So for example if I push an image to a hosted repository:
docker push my-registry.com:5000/myimage:latest
The proxy registry looks for the image as:
docker pull my-registry.com:5000/library/myimage:latest
Which of course doesn't exist. (It would be really great if Nexus would add the /library namespace automatically on image push, or at least make this a configurable option at the repo level).
If you were to do the following:
docker push my-registry:5000/library/myimage:latest
or even:
docker push my-registry:5000/mynamespace/myimage:latest
The your proxy will be able to find the image.
docker pull my-proxy-registry:5000/mynamespace/myimage:latest
I was developing a Custom plugin for Kong.
To start off I followed guidelines listed in this tutorial
http://streamdata.io/blog/developing-an-helloworld-kong-plugin/
Few changes that I made along the way were changing dependency in the rockspec file for "lrexlib-pcre" from version 2.8.0-1 to 2.7.2-1 due to compilation problems that I faced with 2.8.0-1 version.
Please note that I am working in the next branch. The master branch has version 2.7.2-1 listed.
The tutorial assumes Kong version 0.4.2-1 while I am working with Kong version 0.5.2-1.
I have listed my plugin in kong.yml. Last listed is helloworld plugin
plugins_available:
- ssl
- jwt
- acl
- cors
- oauth2
- tcp-log
- udp-log
- file-log
- http-log
- key-auth
- hmac-auth
- basic-auth
- ip-restriction
- mashape-analytics
- request-transformer
- response-transformer
- request-size-limiting
- rate-limiting
- response-ratelimiting
- helloworld
I have listed the helloworld files in rockspec file at the last.
["kong.plugins.helloworld.handler"] =
"kong/plugins/helloworld/handler.lua",
["kong.plugins.helloworld.access"] = "kong/plugins/helloworld/access.lua",
["kong.plugins.helloworld.schema"] = "kong/plugins/helloworld/schema.lua"
Compilation is successful but kong refuses to list helloworld plugin as available in the node. All other builtin plugins are shown as available in the server
I tried enabling the plugin anyway with mock api. It doesn't work as expected and trying to restart kong lists error
nginx: [error] [lua] init_by_lua:5: Startup error:
/usr/local/share/lua/5.1/kong.lua:82: You are using a plugin that has
not been enabled in the configuration: helloworld [INFO] dnsmasq
stopped [ERR] Could not start Kong
I know there were some breaking changes introduced in Kong version 0.5. I followed the changelog, but I found nothing that would help.
Am i missing a setting a configuration somewhere?
Any help would be appreciated.
Try the following in your kong.yml:
custom_plugins:
- helloworld
I fixed this issue by adding things in custom_plugins and lua_package_path.
Here are the steps to enable and use custom plugin in kong Env.
1 - Add custom plugin name in --- custom_plugins = hello-world
2 - Install hello-world plugin by using following steps ---
If you have source code of your plugin then move into it and execute the command --- "luarocks make"
it will install your plugin.
now you have to execute a command "make install-dev" make sure your plugin have makefile like as --
Once you execute this command "make install-dev". it will create lua file at a location something like that -
/your-plugin-path/lua_modules/share/lua/5.1/kong/plugins/your-plugin-name/?.lua
just copy this path and add it into the kong configuration file in lua_package_path
Something like that --
lua_package_path=/your-plugin-path/lua_modules/share/lua/5.1/kong/plugins/your-plugin-name/?.lua
Now you done your job.
Just start kong -- kong start --vv
You will see that the plugin loaded into kong plugin env.
#Enjoy
I am trying to create an OpenShift application using the --from-code option to grab the application code from GitHub. I've created two different OpenShift QuickStarts -- with one, the --from-code option works, and with the other, it doesn't work.
So clearly I'm doing something wrong in the QuickStart that isn't working. But I can't see what I'm doing wrong. I either get error 504 or an error occurred, neither of which tells me what the problem is, and there doesn't seem to be a verbose flag to get more details on the error.
Tests-Mac:~ testuser$ rhc app create sonr diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart --from-code https://github.com/citrusbyte/SONR.git
The cartridge 'http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart' will be downloaded and installed
Application Options
-------------------
Domain: schof
Cartridges: diy-0.1, http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
Source Code: https://github.com/citrusbyte/SONR.git
Gear Size: default
Scaling: no
Creating application 'sonr' ... Server returned an unexpected error code: 504
Tests-Mac:~ testuser$ rhc app create sonr diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart --from-code https://github.com/citrusbyte/SONR.git
The cartridge 'http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart' will be downloaded and installed
Application Options
-------------------
Domain: schof
Cartridges: diy-0.1, http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
Source Code: https://github.com/citrusbyte/SONR.git
Gear Size: default
Scaling: no
Creating application 'sonr' ...
An error occurred while communicating with the server. This problem may only be temporary. Check that you have correctly specified your
OpenShift server 'https://openshift.redhat.com/broker/rest/domain/schof/applications'.
Tests-Mac:~ testuser$
That's creating an application with --from-code using this repo: https://github.com/citrusbyte/SONR . If I use this repo it works flawlessly: https://github.com/citrusbyte/openshift-sinatra-redis
The code itself seems to be good, as I can create an empty new application, merge the SONR code in, and it works flawlessly.
What am I doing wrong?
UPDATE: I've worked around this issue by creating the app in two stages instead of doing it in one stage:
rhc app create APPNAME diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
cd APPNAME
git remote add github -f https://github.com/citrusbyte/SONR.git
git merge github/master -s recursive -X theirs
git push origin master
I'd still love to know why doing it in one step was failing, though.
#developercorey had the right idea.
I tried with a ridiculous timeout of 99999, and then got a different timeout error that I don't think I can change:
$ rhc app create APPNAME diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart --from-code https://github.com/citrusbyte/SONR.git --timeout 99999
...
Creating application 'APPNAME' ...
The initial build for the application failed: Shell command '/sbin/runuser -s /bin/sh 5328a9385973ca70150002af -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c5,c974' /bin/sh -c \"gear postreceive --init >> /tmp/initial-build.log 2>&1\""' exceeded timeout of 229
The fix I mentioned in my earlier update is working perfectly, and that's what I recommend anyone with a similar problem try -- I'm creating the app as empty without the --from-code option, and then merging in the code I wanted to use in a separate step:
rhc app create APPNAME diy-0.1 http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart
cd APPNAME
git remote add github -f https://github.com/citrusbyte/SONR.git
git merge github/master -s recursive -X theirs
git push origin master
It could be that the application takes to long to clone/setup, and the creation is timing out. Something you can try is to create the application without the --from-code, then clone it locally, and merge in your code from github, then do a git push. This operation has a much longer timeout period, and will also let you see what, if any, errors that you get since the application won't disappear if it doesn't succeed, unlike an app create.