I have Artifactory Pro (trial) installed on VM in Azure. I have to check how will it work without internet access. I use only Maven repositories so I created NSG outbound role for blocking HTTPS access. After reboot Artifactory won't load. From log:
2023-01-04T11:27:40.641Z [jfrou] [WARN ] [0e5f4bf9d6dc7f57] [local_topology.go:274 ] [main ] [] - Readiness test failed with the following error: "required node services are missing or unhealthy"
2023-01-04T11:27:41.087Z 35[jfob ] [INFO ] [44efcd4b59cba041] [artifactory_client.go:159 ] [main ] [] - Fetching Artifactory version [artifactory_client]
2023-01-04T11:27:41.092Z 35[jfob ] [INFO ] [44efcd4b59cba041] [artifactory_client.go:176 ] [main ] [] - Got HTTP 503 from Artifactory [artifactory_client]
2023-01-04T11:27:41.092Z 35[jfob ] [INFO ] [44efcd4b59cba041] [artifactory_client.go:183 ] [main ] [] - Fetch base URL from Artifactory returned: 503 <!--
~ Artifactory is a binaries repository manager.
When a team member restores nuget packages using donet, artifactory enters a loop with the errors below until the memory overflows.
The problem only happens when he adds in his Nuget.Config the tag protocolVersion="3"
<add key="Company" value="https://repo.company.com/artifactory/api/nuget/v3/rdi-nuget-virtual" protocolVersion="3" />
In the tests we noticed that the use of the protocolversion=3 tag causes this.
This tag is necessary because the download of the file is much faster
Anyone have an idea what it could be?
Could you help me please?
logs below
Version jfrog: 7.29.8 rev 72908900
2022-01-11T14:00:57.263Z [jfrt ] [WARN ] [6cdecc92eaf486c7] [.r.ArtifactoryResponseBase:136] [ttp-nio-8081-exec-43] - Sending HTTP error code 403: Download request for repo:path 'nuget-remote-cache:.nuGetV3/feed.json' is forbidden for user: 'lcunha'.
2022-01-11T14:00:57.264Z [jfrt ] [ERROR] [6cdecc92eaf486c7] [etV3VirtualAndRemoteCommon:274] [ttp-nio-8081-exec-43] - Failed to download resource in repo: nuget-remote, at url: https://api.nuget.org/v3/index.json. HTTP STATUS CODE: 403
2022-01-11T14:00:57.264Z [jfrt ] [ERROR] [6cdecc92eaf486c7] [etV3VirtualAndRemoteCommon:133] [ttp-nio-8081-exec-43] - Failed to convert artifactory url (https://repostaging.companysoftware.com:443/artifactory/api/nuget/v3/company-nuget-virtual/registration-semver2) to original remote url for repo: nuget-remote, package: xunit.core
java.lang.NullPointerException: null
at java.base/java.util.Objects.requireNonNull(Objects.java:221)
2022-01-11T14:01:56.786Z [jfrou] [ERROR] [2016c910242342cc] [external_topology.go:82 ] [main ] - Failed fetching external topology from Access: Get "http://localhost:8040/access/api/v1/topology": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2022-01-11T14:02:04.871Z [jfrou] [WARN ] [7f3fb3a30ade9665] [local_topology.go:268 ] [main ] - Readiness test failed with the following error: "required node services are missing or unhealthy"
2022-01-11T14:02:09.877Z [jfrou] [ERROR] [7f3fb3a30ade9665] [local_topology.go:128 ] [main ] - periodic send heartbeat failed for 4 consecutive times. Last error: failed sending heartbeat information to Access: failed closing Access grpc client: closing heartbeat client and waiting for response timed-out
java.lang.OutOfMemoryError: Java heap space
-XX:OnOutOfMemoryError="kill -9 %p"
Executing /bin/sh -c "kill -9 3974"...
2022-01-11T14:02:11.185Z [jfrou] [WARN ] [5a89519a8048b91d] [local_topology.go:268 ] [main ] - Readiness test failed with the following error: "required node services are missing or unhealthy"
2022-01-11T14:02:11.196Z [jfrou] [ERROR] [79bb63bc55c1ed15] [external_topology.go:82 ] [main ] - Failed fetching external topology from Access: Get "http://localhost:8040/access/api/v1/topology": read tcp 127.0.0.1:55970-127.0.0.1:8040: read: connection reset by peer
2022/01/11 14:02:11 httputil: ReverseProxy read error during body copy: read tcp 127.0.0.1:56788->127.0.0.1:8045: read: connection reset by peer
2022/01/11 14:02:11 httputil: ReverseProxy read error during body copy: read tcp 127.0.0.1:56788->127.0.0.1:8045: read: connection reset by peer
2022/01/11 14:02:11 httputil: ReverseProxy read error during body copy: read tcp 127.0.0.1:56788->127.0.0.1:8045: read: connection reset by peer
2022-01-11T14:02:11.208Z 35[jfob ] [WARN ] [1ed879c85a5af005] [access_join.go:70 ] [main ] - Refreshing platform config change events gRPC stream - target server is unavailable - if issue persists check communication with access [access_client]
/opt/jfrog/artifactory/app/bin/artifactory.sh: line 359: 3974 Killed $TOMCAT_HOME/bin/catalina.sh run
Can you share your setup?
-The application server is the CentOS Linux release 7.9.2009 (Core)
-MySQL as backend Database
-Also, we are using the Apache as reverse proxy to apply the SSL certificate.
How are you running Artifactory?
-The Artifactory is container based: releases-docker.jfrog.io/jfrog/artifactory-pro:7.29.8
How much resources to you give it? Memory and CPU
The server has allocated:
2 vCPU
16 GB RAM
SSD
What are the java memory settings? Mostly interested in heap settings (Xms and Xmx)?
We are using the standard configuration. No changes were made in the Xms and Xms parameters.
As you are using Artifactory v7.x? You may refer to our System Requirements wiki page for the recommended hardware based on your environment.
Also, from on the error message shared, it looks like the user does not have proper permissions hence we see 403 errors. Please do validate and assign the required permissions to the user and let us know the results.
Sending HTTP error code 403:
Download request for repo:
path 'nuget-remote-cache:.nuGetV3/feed.json' is forbidden for user: 'lcunha'.
We just update our artifactory 6.10.9 to 7.6.3 on Ubuntu using debian package.
The migration.log show that the migration complete successfully
the service start bu we get a 404 when accessing the server.
There is errors on router-service.log:
[jfrou] [FATAL] [1ef8ac7a8bab9898] [bootstrap.go:93] [main] - Cluster join: Failed resolving join key: failed resolving 'shared.security.joinKey' key; file does not exist: /opt/jfrog/artifactory/var/etc/security/join.key
in frontend-service.log :
[jffe ] [ERROR] [ ] [ ] [main] - Error: connect ECONNREFUSED 127.0.0.1:8046
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
in metadata-service.log :
[jffe ] [ERROR] [ ] [ ] [main] - Error: connect ECONNREFUSED 127.0.0.1:8046
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
And this is the first errors in the console.log :
2020-08-18T19:27:12.494Z [jfrt ] [ERROR] [d2fc2e16d5fc1039] [actoryContextConfigListener:83] [ocalhost-startStop-1] - Failed initializing Artifactory context: Artifactory home not initialized.
2020-08-18T15:27:12.495L [tomct] [SEVERE] [ ] [org.apache.catalina.core.StandardContext] [org.apache.catalina.core.StandardContext startInternal] - One or more listeners failed to start. Full details will be found in the appropriate container log file
2020-08-18T15:27:12.497L [tomct] [SEVERE] [ ] [org.apache.catalina.core.StandardContext] [org.apache.catalina.core.StandardContext startInternal] - Context [/artifactory] startup failed due to previous errors
Does someone already had this kind of issue ? Thanks for clues.
Based on the error log snippet, it looks like the issue is with the router not coming up. Can you check the $JFROG_HOME/artifactory/var/log/router-service.log file if there is an error entry causing this issue?
Final problem was due to bad host file. fixing it with good entries solved the last problem
I upgraded my Artifactory Pro server from version 6.x to 7.x.
I have ldap authentication with MS ActiveDirectory After upgrade some users cannot login to the web-gui with 500 error. I founded error in artifactory logs:
2020-07-17T07:50:44.288Z [jffe ] [ERROR] [6261074d6e4d564 ] [ ] [main ] - http request failed with code - BAD_REQUEST
2020-07-17T07:50:44.288Z [jffe ] [ERROR] [6261074d6e4d564 ] [ ] [main ] - http request failed with message - Could not find user 'ad-test-usr'
2020-07-17T07:50:44.289Z [jffe ] [ERROR] [ ] [ ] [main ] - Error: Request failed with status code 400
at createError (/opt/jfrog/artifactory/app/frontend/bin/server/dist/node_modules/#jfrog/access-nodejs-client/node_modules/axios/lib/core/createError.js:16:15)
at settle (/opt/jfrog/artifactory/app/frontend/bin/server/dist/node_modules/#jfrog/access-nodejs-client/node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (/opt/jfrog/artifactory/app/frontend/bin/server/dist/node_modules/#jfrog/access-nodejs-client/node_modules/axios/lib/adapters/http.js:237:11)
at IncomingMessage.emit (events.js:203:15)
at endReadableNT (_stream_readable.js:1145:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
But user exist in AD and in Artifactory.
It was a bug, in version 6.x logins are case insensitive “UserID” and “userid” are valid both and I can login with any of them.
In version 7.x login is case sensitive and valid is only with lower case “userid”
When I download, unpack and launch artifactory.sh I see the following error
2020-02-26T21:32:50.496Z [jfac ] [ERROR] [c1b4de79a3f11666] [.j.a.s.s.r.JoinServiceImpl:253] [http-nio-8040-exec-1] - Could not validate router Check-url: http://XXXXXX:8082/router/api/v1/system/ping
And then
2020-02-26T21:32:55.636Z [jfac ] [WARN ] [67b9d42698f5614c] [o.j.c.ExecutionUtils:141 ] [pool-6-thread-2 ] - Retry 20 Elapsed 9.04 secs failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again
I realize I'm missing something obvious but couldn't figure it out yet. Any suggestions? Thanks.
Alexey, I suspect ipv6 ip being picked up by the start-script, causing this.
Can you update your system.yaml (will be in var/etc/ folder) with the following and try?
shared:
node:
ip: <your ipv4 IP>
Might be helpful to anyone. I had tried many things mentioned on google to solve the issue but does not work. Finally, it gets resolved by setting up proper resources. I used 4 GB RAM and 2 core and it worked
Alexey, Do you see any error in:
tomcat localhost log - will be at var/log/tomcat
router_service.log - will be at var/log
access_service.log - will be at var/log
Potential issue may be -
your box is not allowing localhost calls (due to some security set-up), or
all dependent services are not starting for some other reason
Also - please check which script are you using, there is a new artifactory.sh script packed in app/bin folder, which should be used.
In my case, I had a similar error the moment i was trying to access artifactory. Logs of router-service:
[root#artifactory-master log]# tail -f router-service.log
2020-03-20T22:17:05.328Z [jfrou] [INFO ] [ ] [bootstrap.go:70 ] [main ] - Router (jfrou) service initialization started. Version: 1.1.0 Revision: c2646fcb28e2d4ca095b07aacebe509d934cef77 PID: 19062 Home: /opt/jfrog/artifactory
2020-03-20T22:17:05.329Z [jfrou] [INFO ] [ ] [bootstrap.go:73 ] [main ] - JFrog Router IP: ::1
2020-03-20T22:17:05.334Z [jfrou] [INFO ] [ ] [bootstrap.go:159 ] [main ] - System configuration encryption report:
shared.newrelic.licenseKey: does not exist in the config file
shared.security.joinKeyFile: file '/opt/jfrog/artifactory/var/etc/security/join.key' - already encrypted
2020-03-20T22:17:05.336Z [jfrou] [INFO ] [ ] [bootstrap.go:78 ] [main ] - JFrog Router Service ID: jfrou#01e3wgemz9esckmd8v48etdy18
2020-03-20T22:17:05.336Z [jfrou] [INFO ] [ ] [bootstrap.go:79 ] [main ] - JFrog Router Node ID: artifactory-master
2020-03-20T22:17:07.354Z [jfrou] [INFO ] [ ] [config_holder.go:107 ] [main ] - configuration update detected
2020-03-20T22:17:10.738Z [jfrou] [FATAL] [ ] [bootstrap.go:100 ] [main ] - Cluster join: Failed joining the cluster; Error: Error response from service registry, status code: 400; message: Could not validate router Check-url: http://::1:8082/router/api/v1/system/ping; detail: I/O error on GET request for "http:///:1:8082/router/api/v1/system/ping": URI does not specify a valid host name: http:///:1:8082/router/api/v1/system/ping; nested exception is org.apache.http.client.ClientProtocolException: URI does not specify a valid host name: http:///:1:8082/router/api/v1/system/ping
To give you some context, i am running artifactory in a centos 8 VM and I'm accessing artifactory graphical interface via windows machine.
That means, I am using a web browser (Chrome) to navigate to the artifactory instance.
For that, in the VM side i updated the files "hosts" and "hostname" (/etc/):
hosts:
127.0.0.1 localhost artifactory-master
::1 localhost artifactory-master
hostname:
artifactory-master
In the windows machine, i updated the hosts file located in "C:\Windows\System32\drivers\etc" with the VM host ip and hostname:
# localhost name resolution is handled within DNS itself.
# 127.0.0.1 localhost
# ::1 localhost
192.100.100.10 artifactory-master
(in order to get the ip of the VM machine run the command ifconfig)
Then, I started artifactory by running the command:
service artifactory start
And tried to access artifactory in the browser unsucessfuly:
http://artifactory-master:8082/ui
I stopped the service and in order to accomplish my goal after some tryouts, I realized that I had to comment out the address "::1" in the "hosts" file (/etc/):
hosts:
127.0.0.1 localhost artifactory-master
#::1 localhost artifactory-master
Finally, I started the service again and i was able to access artifactory. Logs of router-service :
2020-03-24T23:02:17.219Z [jfrou] [INFO ] [ ] [bootstrap.go:70 ] [main ] - Router (jfrou) service initialization started. Version: 1.1.0 Revision: c2646fcb28e2d4ca095b07aacebe509d934cef77 PID: 14542 Home: /opt/jfrog/artifactory
2020-03-24T23:02:17.220Z [jfrou] [INFO ] [ ] [bootstrap.go:73 ] [main ] - JFrog Router IP: 127.0.0.1
2020-03-24T23:02:17.224Z [jfrou] [INFO ] [ ] [bootstrap.go:159 ] [main ] - System configuration encryption report:
shared.newrelic.licenseKey: does not exist in the config file
shared.security.joinKeyFile: file '/opt/jfrog/artifactory/var/etc/security/join.key' - already encrypted
2020-03-24T23:02:17.227Z [jfrou] [INFO ] [ ] [bootstrap.go:78 ] [main ] - JFrog Router Service ID: jfrou#01e3wgemz9esckmd8v48etdy18
2020-03-24T23:02:17.227Z [jfrou] [INFO ] [ ] [bootstrap.go:79 ] [main ] - JFrog Router Node ID: artifactory-master
2020-03-24T23:02:19.572Z [jfrou] [INFO ] [ ] [config_holder.go:107 ] [main ] - configuration update detected
2020-03-24T23:02:25.663Z [jfrou] [INFO ] [ ] [join_executor.go:180 ] [main ] - Cluster join: Successfully joined the cluster
2020-03-24T23:02:25.813Z [jfrou] [INFO ] [ ] [registry_handler.go:89 ] [main ] - the following services were registered automatically based on persisted data: jfac#01e3wgdn6q0gvj0czswc8k0gp8, jffe#000, jfmd#01e3wges9tvwawj403y5mxfjp7, jfrt#01e3wgfass87mh1nbcv5rv1t98
2020-03-24T23:02:25.984Z [jfrou] [INFO ] [ ] [main.go:36 ] [main ] - Router (jfrou) service initialization completed in 8.808 seconds. Listening on port: 8082
2020-03-24T23:03:01.281Z [jfrou] [INFO ] [7e7df2f621a4e1aa] [local_topology.go:212 ] [main ] -
###############################################################
### All services started successfully in 44.081 seconds ###
###############################################################
PS: my artifactory version is the OSS 7.2.1
we have put the IP in our node, and this doesnt work.
we can see that the router is still using LOCALHOST and that its using the ::1 IPV6, despite our system.yaml being indented correctly.
this was working fine (running system for more than 4 months) and then the most recent update this started to fail.
Does anyone have anything better than "check the file" - that actually addresses the issue, which is the following:
OS's generally come with both localhost set to the 127.0.0.1 and the ::1 to localhost.
JFROG router is "dumb " in that its picking up the url https://localhost:8046, but then trying to do resolution to the ::1 (catch all for IPv6).
many SO ticket shows #prasanna and others doing "edits" to the file, but even with correct syntax, we can see JFROG's product is not doing what is says its doing.
example of system.yaml (you can see the the indents are correct and in fact, this is from the system.yaml-full-example template that JFROG provides.
SHARED CONFIGURATIONS
A shared section for keys across all services in this config
shared:
## Security Configuration
security:
## Join key value for joining the cluster (takes precedence over 'joinKeyFile')
#joinKey: ""
## Join key file location
#joinKeyFile: "<For example: JFROG_HOME/artifactory/var/etc/security/join.key>"
## Master key file location
## Generated by the product on first startup if not provided
#masterKeyFile: "<For example: JFROG_HOME/artifactory/var/etc/security/master.key>"
## Maximum time to wait for key files (master.key and join.key)
#bootstrapKeysReadTimeoutSecs: 120
## Node Settings
node:
## A unique id to identify this node.
## Default auto generated at startup.
id: "art-00"
## Default auto resolved by startup script
ip: 10.x.34.63 (x is there on purpose)
## Sets this node as primary in HA installation
you can see plainly inside the logs when you start what is happening as the OP showed.
SAMPLE LOG