docker pull manifest unknown blob errors - nginx

I am currently trying to set up nexus as a private registry for docker images and have been able to login,push,pull,search through the nexus repository .
Currently we are stuck with pulling images from docker if not available within the nexus hosted repository . We are facing errors ranging from blob unknown to manifest unknown.
[root#server1446 ~]$ docker pull server908.int.org.com:6666/centos
Using default tag: latest
Trying to pull repository server908.int.org.com:6666/centos ...
manifest unknown: manifest unknown
Tried following reference [Setup-Docker-Private-Registry-in-Nexus-Repository-OSS-3.0.0][1]
[1]: https://github.com/TerrenceMiao/nexus/wiki/Setup-Docker-Private-Registry-in-Nexus-Repository-OSS-3.0.0 to set up the docker(proxy) and docker(hosted) repo with docker(group) but it throws blob unknown to registry error.
docker(hosted) is configured with http port 4444 & docker(group) with http port 5555 and we have used the same within nginx configuration as below but neither seems to be working .
server {
listen 6666;
server_name server908.int.org.com;
keepalive_timeout 60;
ssl on;
ssl_certificate /etc/ssl/certs/orgnexus.crt;
ssl_certificate_key /etc/ssl/certs/orgnexus.key;
ssl_ciphers HIGH:!kEDH:!ADH:!MD5:#STRENGTH;
ssl_session_cache shared:TLSSSL:16m;
ssl_session_timeout 10m;
ssl_prefer_server_ciphers on;
client_max_body_size 1G;
chunked_transfer_encoding on;
location / {
access_log /var/log/nginx/docker.log;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
proxy_pass http://server908.int.org.com:5555;
proxy_read_timeout 90;
}
We have commented below entries in "/etc/sysconfig/docker" file .
http_proxy=http://x.x.x.x:3128
https_proxy=http://x.x.x.x:3128

Below are my configurations that got it working .
server {
proxy_send_timeout 120;
proxy_read_timeout 300;
proxy_buffering off;
tcp_nodelay on;
server_tokens off;
client_max_body_size 1G;
listen 80;
server_name box.company.net;
location / {
rewrite ^(.*) https://box.company.net$1 301;
}
}
server {
listen 443;
server_name box.company.net;
keepalive_timeout 60;
ssl on;
ssl_certificate /etc/ssl/certs/ssl.crt;
ssl_certificate_key /etc/ssl/certs/ssl.key;
ssl_ciphers HIGH:!kEDH:!ADH:!MD5:#STRENGTH;
ssl_session_cache shared:TLSSSL:16m;
ssl_session_timeout 10m;
ssl_prefer_server_ciphers on;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
proxy_pass http://box.company.net:8081;
proxy_read_timeout 90;
}
}
# correlates to your nexus http connector
server {
listen 6666;
server_name box.company.net;
keepalive_timeout 60;
ssl on;
ssl_certificate /etc/ssl/certs/ssl.crt;
ssl_certificate_key /etc/ssl/certs/ssl.key;
ssl_ciphers HIGH:!kEDH:!ADH:!MD5:#STRENGTH;
ssl_session_cache shared:TLSSSL:16m;
ssl_session_timeout 10m;
ssl_prefer_server_ciphers on;
client_max_body_size 1G;
chunked_transfer_encoding on;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
if ($request_method !~* GET) {
proxy_pass http://box.company.net:4444;
}
if ($request_method = GET) {
proxy_pass http://box.company.net:5555;
}
proxy_read_timeout 90;
}
}
Commented below entries with "/etc/default/docker" file .
http_proxy=http://x.x.x.x:3128
https_proxy=http://x.x.x.x:3128
Restart Nginx.
Perform Login
[test#server ~]$ docker login -u admin -p admin123 box.company.net:6666
Login Succeeded
Post login a file name "config.json" will be created under ".docker" directory
[test#server ~]$ cat ~/.docker/config.json
{
"auths": {
"box.company.net:6666": {
"auth": "YWRtaW46YWRtaW4xMjM="
}
}
}
Search for Images available in docker hub.
[test#server ~]$ docker search box.company.net:6666/ubuntu
INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED
company.net box.company.net:6666/ubuntu Ubuntu is a Debian-based Linux operating s... 6186 [OK]
Pull the images from docker hub via nexus proxy.
[test#server ~]$ docker pull box.company.net:6666/ubuntu
Using default tag: latest
Trying to pull repository box.company.net:6666/ubuntu ...
sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4: Pulling from box.company.net:6666/ubuntu
75c416ea735c: Pull complete
c6ff40b6d658: Pull complete
a7050fc1f338: Pull complete
f0ffb5cf6ba9: Pull complete
be232718519c: Pull complete
Digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4
Status: Downloaded newer image for box.company.net:6666/ubuntu:latest
Tag The pulled images
docker tag box.company.net:6666/ubuntu:latest box.company.net:6666/ubuntu:1
Push to NexusHostedRepo (Port : 4444)
[test#server ~]$ docker push box.company.net:6666/ubuntu:1
The push refers to a repository [box.company.net:6666/ubuntu]
0566c118947e: Pushed
6f9cf951edf5: Pushed
182d2a55830d: Pushed
latest: digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4 size: 1357
Pull from Nexus Repo (This should be quick than the pull from docker hub )
[test#server ~]$ docker pull box.company.net:6666/ubuntu:1
Trying to pull repository box.company.net:6666/ubuntu ...
sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4: Pulling from server908.int.org.com:6666/ubuntu
75c416ea735c: Pull complete
c6ff40b6d658: Pull complete
a7050fc1f338: Pull complete
Digest: sha256:a0ee7647e24c8494f1cf6b94f1a3cd127f423268293c25d924fbe18fd82db5a4
Status: Downloaded newer image for box.company.net:6666/ubuntu:1
--------------------------------------------------------------------------------
Also make sure you add you proxy server certificate within Nexus SSL Certificate Section .
keytool -J-Dhttps.proxyHost=<proxy_hostname> -J-Dhttps.proxyPort=<proxy_port> -printcert -rfc -sslserver <remote_host_name:remote_ssl_port>
Replace proxy_hostname and proxy_port with the HTTP proxy server that Nexus is configured with under Administration -> Server. Replace remote_host_name:remote_ssl_port with one of the remote host and port having the certification problem. You can omit the port if it is the default 443. For docker it will be registry-1.docker.io:443
You should see at least two entries printed by the above command. Take the last certificate content printed and copy it entirely to your clipboard. This should be your proxy server's certificate, added to the end of the certificate chain.
The copied certificate content should start with -----BEGIN CERTIFICATE----- and end with-----END CERTIFICATE-----.
Then in the Nexus UI, go to Administration -> SSL Certificates and click Add... and choose Paste PEM. Paste the certificate contents to the dialog that opens.
Click Load Certificate. Verify the certificate contents in the next window. Verify the Issuer details listed are from your proxy server certificate. When you are satisfied click Add Certificate.
Hope this helps.

Related

Running Apache OpenMeetings with Nginx Reverse Proxy?

I am trying to install Apache OpenMeetings. I however wants to use Nginx as the reverse proxy to run the application on port 443 using Let's Encrypt free SSL.
If I try to load the application on port 5080, I successfully get the interface, but when try using the domain name on port 443 HTTPS, It is not loading the resources.
Image with Errors.
Here's my nginx virtual host file.
upstream openmeetings {
server 127.0.0.1:5080;
}
server {
listen 80;
server_name openmeetings.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443;
server_name openmeetings.example.com;
ssl_certificate /etc/letsencrypt/live/openmeetings.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/openmeetings.example.com/privkey.pem;
ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/openmeetings.access.log;
location / {
proxy_pass http://openmeetings;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_redirect off;
}
}
I faced same problem. (vit Openmeetings 5.0.0-M4)
I found next:
Openmeetings use ajax over WebSocket.
adding
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
to http section
and
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
to location
It solve status 400 problem
Then I meet problem with Content Security Policy
I feel like connect-src policy configured automatic on first connect to server.
So after change used domain I need restart Openmeetings.
Problem with media stream play
On Check setup recording produce long browser console message ending with
onaddstream is deprecated! Use peerConnection.ontrack instead.
...
Remote ICE candidate received
Look like it incompatibility with old Firefox 54.0 on Linux
On latest Firefox 75.0 on Windows it works!
It is also necessary to rewrite server.xml referring to
nginx managed SSL with Tomcat 7
<Valve className="org.apache.catalina.valves.RemoteIpValve"
remoteIpHeader="x-forwarded-for"
remoteIpProxiesHeader="x-forwarded-by"
protocolHeader="x-forwarded-proto"
/>

Nginx Sub domain setup

I'm trying to setup Nginx so I can have sub domains like
www.MySite.com - Main website (Works correctly)
jenkins.MySite.com - sub domain for Jenkins
gitlab.MySite.com - sub domain for Gitlab
I've tried following various tutorials and I seem to have included everything required to make this work, but still to no avail.
I've followed this: https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-with-ssl-as-a-reverse-proxy-for-jenkins
and various other sources online.
[Nginx Server Block]
I've edited my nginx.conf file, I've created a new nginx/sites-available conf file for Jenkins and symlinked it to sites-enabled.
This is my default jenkins JENKINS_ARGS
JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpListenAddress=127.0.0.1 --httpPort=$HTTP_PORT -ajp13Port=$AJP_PORT"
This is an example of my jenkins server block in nginx
server
{
listen 80;
return 301 https://$host$request_uri;
}
server
{
listen 443;
server_name jenkins.MySite.com;
#ssl_certificate /etc/nginx/cert.crt;
#ssl_certificate_key /etc/nginx/cert.key;
#ssl on;
#ssl_session_cache builtin:1000 shared:SSL:10m;
#ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
#ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
#ssl_prefer_server_ciphers on;
access_log /var/log/nginx/jenkins/access.log;
location /
{
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Fix the “It appears that your reverse proxy set up is broken" error.
proxy_pass http://127.0.0.1:8080;
proxy_read_timeout 90;
proxy_redirect http://127.0.0.1:8080 https://jenkins.MySite.com;
}
}
I've also created an A record in DigitalOcean - Network
and also a CNAME
Much help would be appreciated.
Thanks
All these 3-setups need separate ngnix config files and supervirosor files as you did for main site. make soft link of those files and put them in respective etc/nginx/sites-avai and sites-enable and also soft link the supervisor files to etc/supervisor/conf.d
To check whether the nginx file is properly configured, you need to test it.
sudo nginx -t

Is it bad practice to have multiple Unicorn App Servers with one Nginx reverse proxy server?

I have a Linux box running Ubuntu 14.04 with about 50gb of memory.
I've got a 5 or 6 Ruby-on-Rails web applications, each with a Unicorn App server, all served by an Nginx reverse proxy server.
Each app is hosted in a sub-directory.
eg:
www.webserver.com/app1
www.webserver.com/app2
Each app gets maybe 50-100 requests per day. They are all little apps to facilitate business processes at my firm.
My Nginx config file looks something like this:
upstream app1 {
#path to Unicorn SOCK file;
}
upstream app2 {
#path to Unicorn SOCK file;
}
upstream app3 {
#path to Unicorn SOCK file;
}
# ...several more apps
server {
listen 443 ssl;
access_log #path;
error_log #path;
ssl_certificate #path;
ssl_certificate_key #path;
add_header X-UA-Compatible "IE=Edge,chrome=1";
root /srv/apps/app1/public;
location /app1 {
proxy_pass http://app1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
}
location /app2 {
proxy_pass http://app2;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
}
location /app3 {
proxy_pass http://app3;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
}
# ...several more apps
}
This setup has worked without issue for a year or so, but I have this nagging feeling I'm doing this all wrong....
Am I going to run into problems if I keep adding apps? Is there a better way to do this?
Update:
By "problems," I mean:
static resource path collisions?
memory issues? namely, using more than I need to accomplish same behavior?
And by "a better way to do this," I mean:
other than sending requests to the relevant unicorn server by parsing out the name of the sub-directory in the URL
should I be using a single Nginx reverse proxy to serve multiple apps?
For the same configuration into differents apps, you can use include directive.
Example, create file named /etc/nginx/global_proxy.conf with this content :
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
And in nginx.conf your section /appX :
location /appX {
proxy_pass http://appX;
include /etc/nginx/global_proxy.conf;
}
And to increase your security, i recommend you adding dhparam, and add this to SSL configuration :
# SSL :
# drop SSLv3 (POODLE vulnerability)
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# Recommanded ciphers
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
# Diffie–Hellman key exchange (D–H)
ssl_dhparam /etc/nginx/ssl/dhparam.pem;
# config to enable HSTS(HTTP Strict Transport Security)
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# force timeouts if one of backend is died
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
To generate dhparam.pem file :
openssl dhparam -out dhparam.pem 4096

How to create Kubernetes cluster serving its own container with SSL and NGINX

I'm trying to build a Kubernetes cluster with following services inside:
Docker-registry (which will contain my django Docker image)
Nginx listenning both on port 80 and 443
PostgreSQL
Several django applications served with gunicorn
letsencrypt container to generate and automatically renew signed SSL certificates
My problem is a chicken and egg problem that occurs during the creation of the cluster:
My SSL certificates are stored in a secret volume that is generated by the letsencrypt container. To be able to generate the certificate, we need to show we are owner of the domain name and this is done by validating a file is accessible from the server name (basically this consist of Nginx being able to serve a staticfile over port 80)
So here occurs my first problem: To serve the static file needed by letsencrypt, I need to have nginx started. The SSL part of nginx can't be started if the secret hasn't been mounted and the secret is generated only when let's encrypt succeed...
So, a simple solution could be to have 2 Nginx containers: One listening only on port 80 that will be started first, then letsencrypt then we start a second Nginx container listening on port 443
-> This kind of look like a waste of resources in my opinion, but why not.
Now assuming I have 2 nginx containers, I want my Docker Registry to be accessible over https.
So in my nginx configuration, I'll have a docker-registry.conf file looking like:
upstream docker-registry {
server registry:5000;
}
server {
listen 443;
server_name docker.thedivernetwork.net;
# SSL
ssl on;
ssl_certificate /etc/nginx/conf.d/cacert.pem;
ssl_certificate_key /etc/nginx/conf.d/privkey.pem;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
# required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
chunked_transfer_encoding on;
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
# To add basic authentication to v2 use auth_basic setting plus add_header
auth_basic "registry.localhost";
auth_basic_user_file /etc/nginx/conf.d/registry.password;
add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
proxy_pass http://docker-registry;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
The important part is the proxy_pass that redirect toward the registry container.
The problem I'm facing is that my Django Gunicorn server also has its configuration file in the same folder django.conf:
upstream django {
server django:5000;
}
server {
listen 443 ssl;
server_name example.com;
charset utf-8;
ssl on;
ssl_certificate /etc/nginx/conf.d/cacert.pem;
ssl_certificate_key /etc/nginx/conf.d/privkey.pem;
ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
client_max_body_size 20M;
location / {
# checks for static file, if not found proxy to app
try_files $uri #proxy_to_django;
}
location #proxy_to_django {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
#proxy_pass_header Server;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 65;
proxy_read_timeout 65;
proxy_pass http://django;
}
}
So nginx will successfully start only under 3 conditions:
secret is mounted (this could be addressed by splitting Nginx into 2 separate containers)
registry service is started
django service is started
The problem is that django image is pulling its image from the registry service, so we are in a dead-lock situation again.
I didn't mention it but both registry and django have different ServerName so nginx is able to both serve them
The solution I though about it (but it's quite dirty!) would be to reload nginx several time with more and more configurations:
I start docker registry service
I start Nginx with only the registry.conf
I create my django rc and service
I reload nginx with both registry.conf and django.conf
If there was a way to make nginx start ignoring failing configuration, that would probably solve my issues as well.
How can I cleanly achieve this setup?
Thanks for your help
Thibault
Are you using Kubernetes Services for your applications?
With a Service to each of your Pods, you have a proxy for the Pods. Even if the pod is not started, as long as the Service is started nginx will find it when looking it up as the Service has an IP assigned.
So you start the Services, then start nginx and whatever Pod you want in the order you want.

Artifactory pro server behind an nginx frontend

I am trying to set up ssl for our artifactory server. For this I wish to configure nginx as the reverse proxy. So far I have done the following
-- Installed artifactory pro using its docker image
docker run --name artifactory-registry -p 8081:8081 -v $ARTIFACTORY_HOME/data -v $ARTIFACTORY_HOME/logs -v $ARTIFACTORY_HOME/backup -v $ARTIFACTORY_HOME/etc jfrog-docker-reg2.bintray.io/jfrog/artifactory-pro:latest
-- Insatlled nginx using sudo apt-get install nginx
I have the webapp accessible at http://localhost:8081/artifactory/webapp/#/home
and teh following config file under $ARTIFACTORY_HOME/tomcat/conf/server.xml
<Service name="Catalina">
<Connector port="8081"/>
<!-- This is the optional AJP connector -->
<Connector port="8019" protocol="AJP/1.3"/>
<Engine name="Catalina" defaultHost="localhost">
<Host name="localhost" appBase="webapps"/>
</Engine>
</Service>
From sources found online, the following configuration is needed for the nginx to act as a reverse proxy
server {
listen 80;
server_name yourdomain.com;
root /etc/tomcat7/webapps/apple;
proxy_cache one;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://127.0.0.1:8080/;
}
}
What should be the 'server_name' and 'root' in this path? Also how do I test the reverse proxiness of this setup?
Usually you will configure SSL on it's default HTTPs port (443).
Take a look at this page for configuring nginx with SSL.
The server_name is the host from which you will connect to your nginx (from the browser for example). Usually you will have a DNS address from your company (like artifactory.mycompany.com) and you will use that, but if everything is local you can just put localhost instead.
Here is a working SSL configuration on port 443:
server {
listen 443;
server_name artifactory.mycompany.com;
access_log /var/log/nginx/artifactory.access.log;
error_log /var/log/nginx/artifactory.error.log;
ssl on;
ssl_certificate /etc/nginx/ssl/artifactory.crt;
ssl_certificate_key /etc/nginx/ssl/artifactory.key;
ssl_session_timeout 5m;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP;
ssl_prefer_server_ciphers on;
location /artifactory {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://localhost:8081;
proxy_pass_header Server;
proxy_read_timeout 90;
}
}
Simply put your SSL certificates at the configured locations and you are good to go.
Connecting to https://artifactory.mycompany.com from your browser should now work (or https://localhost if you used that for server_name).

Resources