We just update our artifactory 6.10.9 to 7.6.3 on Ubuntu using debian package.
The migration.log show that the migration complete successfully
the service start bu we get a 404 when accessing the server.
There is errors on router-service.log:
[jfrou] [FATAL] [1ef8ac7a8bab9898] [bootstrap.go:93] [main] - Cluster join: Failed resolving join key: failed resolving 'shared.security.joinKey' key; file does not exist: /opt/jfrog/artifactory/var/etc/security/join.key
in frontend-service.log :
[jffe ] [ERROR] [ ] [ ] [main] - Error: connect ECONNREFUSED 127.0.0.1:8046
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
in metadata-service.log :
[jffe ] [ERROR] [ ] [ ] [main] - Error: connect ECONNREFUSED 127.0.0.1:8046
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
And this is the first errors in the console.log :
2020-08-18T19:27:12.494Z [jfrt ] [ERROR] [d2fc2e16d5fc1039] [actoryContextConfigListener:83] [ocalhost-startStop-1] - Failed initializing Artifactory context: Artifactory home not initialized.
2020-08-18T15:27:12.495L [tomct] [SEVERE] [ ] [org.apache.catalina.core.StandardContext] [org.apache.catalina.core.StandardContext startInternal] - One or more listeners failed to start. Full details will be found in the appropriate container log file
2020-08-18T15:27:12.497L [tomct] [SEVERE] [ ] [org.apache.catalina.core.StandardContext] [org.apache.catalina.core.StandardContext startInternal] - Context [/artifactory] startup failed due to previous errors
Does someone already had this kind of issue ? Thanks for clues.
Based on the error log snippet, it looks like the issue is with the router not coming up. Can you check the $JFROG_HOME/artifactory/var/log/router-service.log file if there is an error entry causing this issue?
Final problem was due to bad host file. fixing it with good entries solved the last problem
Related
When a team member restores nuget packages using donet, artifactory enters a loop with the errors below until the memory overflows.
The problem only happens when he adds in his Nuget.Config the tag protocolVersion="3"
<add key="Company" value="https://repo.company.com/artifactory/api/nuget/v3/rdi-nuget-virtual" protocolVersion="3" />
In the tests we noticed that the use of the protocolversion=3 tag causes this.
This tag is necessary because the download of the file is much faster
Anyone have an idea what it could be?
Could you help me please?
logs below
Version jfrog: 7.29.8 rev 72908900
2022-01-11T14:00:57.263Z [jfrt ] [WARN ] [6cdecc92eaf486c7] [.r.ArtifactoryResponseBase:136] [ttp-nio-8081-exec-43] - Sending HTTP error code 403: Download request for repo:path 'nuget-remote-cache:.nuGetV3/feed.json' is forbidden for user: 'lcunha'.
2022-01-11T14:00:57.264Z [jfrt ] [ERROR] [6cdecc92eaf486c7] [etV3VirtualAndRemoteCommon:274] [ttp-nio-8081-exec-43] - Failed to download resource in repo: nuget-remote, at url: https://api.nuget.org/v3/index.json. HTTP STATUS CODE: 403
2022-01-11T14:00:57.264Z [jfrt ] [ERROR] [6cdecc92eaf486c7] [etV3VirtualAndRemoteCommon:133] [ttp-nio-8081-exec-43] - Failed to convert artifactory url (https://repostaging.companysoftware.com:443/artifactory/api/nuget/v3/company-nuget-virtual/registration-semver2) to original remote url for repo: nuget-remote, package: xunit.core
java.lang.NullPointerException: null
at java.base/java.util.Objects.requireNonNull(Objects.java:221)
2022-01-11T14:01:56.786Z [jfrou] [ERROR] [2016c910242342cc] [external_topology.go:82 ] [main ] - Failed fetching external topology from Access: Get "http://localhost:8040/access/api/v1/topology": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2022-01-11T14:02:04.871Z [jfrou] [WARN ] [7f3fb3a30ade9665] [local_topology.go:268 ] [main ] - Readiness test failed with the following error: "required node services are missing or unhealthy"
2022-01-11T14:02:09.877Z [jfrou] [ERROR] [7f3fb3a30ade9665] [local_topology.go:128 ] [main ] - periodic send heartbeat failed for 4 consecutive times. Last error: failed sending heartbeat information to Access: failed closing Access grpc client: closing heartbeat client and waiting for response timed-out
java.lang.OutOfMemoryError: Java heap space
-XX:OnOutOfMemoryError="kill -9 %p"
Executing /bin/sh -c "kill -9 3974"...
2022-01-11T14:02:11.185Z [jfrou] [WARN ] [5a89519a8048b91d] [local_topology.go:268 ] [main ] - Readiness test failed with the following error: "required node services are missing or unhealthy"
2022-01-11T14:02:11.196Z [jfrou] [ERROR] [79bb63bc55c1ed15] [external_topology.go:82 ] [main ] - Failed fetching external topology from Access: Get "http://localhost:8040/access/api/v1/topology": read tcp 127.0.0.1:55970-127.0.0.1:8040: read: connection reset by peer
2022/01/11 14:02:11 httputil: ReverseProxy read error during body copy: read tcp 127.0.0.1:56788->127.0.0.1:8045: read: connection reset by peer
2022/01/11 14:02:11 httputil: ReverseProxy read error during body copy: read tcp 127.0.0.1:56788->127.0.0.1:8045: read: connection reset by peer
2022/01/11 14:02:11 httputil: ReverseProxy read error during body copy: read tcp 127.0.0.1:56788->127.0.0.1:8045: read: connection reset by peer
2022-01-11T14:02:11.208Z 35[jfob ] [WARN ] [1ed879c85a5af005] [access_join.go:70 ] [main ] - Refreshing platform config change events gRPC stream - target server is unavailable - if issue persists check communication with access [access_client]
/opt/jfrog/artifactory/app/bin/artifactory.sh: line 359: 3974 Killed $TOMCAT_HOME/bin/catalina.sh run
Can you share your setup?
-The application server is the CentOS Linux release 7.9.2009 (Core)
-MySQL as backend Database
-Also, we are using the Apache as reverse proxy to apply the SSL certificate.
How are you running Artifactory?
-The Artifactory is container based: releases-docker.jfrog.io/jfrog/artifactory-pro:7.29.8
How much resources to you give it? Memory and CPU
The server has allocated:
2 vCPU
16 GB RAM
SSD
What are the java memory settings? Mostly interested in heap settings (Xms and Xmx)?
We are using the standard configuration. No changes were made in the Xms and Xms parameters.
As you are using Artifactory v7.x? You may refer to our System Requirements wiki page for the recommended hardware based on your environment.
Also, from on the error message shared, it looks like the user does not have proper permissions hence we see 403 errors. Please do validate and assign the required permissions to the user and let us know the results.
Sending HTTP error code 403:
Download request for repo:
path 'nuget-remote-cache:.nuGetV3/feed.json' is forbidden for user: 'lcunha'.
What reasons could there be that a push of a docker image to an Artifactory repo could crash the entire app?
The symptoms I see are that a layer hangs during the push:
» docker push my-repo/my-app:20211207-afdf6438-test
The push refers to repository [my-repo/my-app]
d4cde41bcf33: Layer already exists
ac14050b2264: Layer already exists
27e5cc646fd0: Pushing [==================================================>] 562.7MB
797b1ec2507f: Layer already exists
34de87854e34: Layer already exists
cde140fcdbee: Layer already exists
be3883e87d34: Layer already exists
9ec86c039eae: Layer already exists
371ce8b24b31: Layer already exists
7e718b9c0c8c: Layer already exists
It hangs on pushing that layer, then Artifactory becomes unresponsive until it finally restarts itself, which takes a while.
When this happens, the logs don't give me much to go on.
2021-12-09T20:06:01.066Z [jfrt ] [INFO ] [44ac995c9c11dc2e] [o.a.e.UploadServiceImpl:465 ] [http-nio-8081-exec-2] - Deploy to 'docker-local:my-app/_uploads/60766841-fac7-4781-b325-da9ef66ba2cf' Content-Length: 1 (estimation) artificial: false
2021-12-09T20:06:27.249Z [jfrt ] [INFO ] [ ] [ffectedConfigStreamObserver:32] [Stream_1639080147248] - publishing full invalidation and attempting to resubscribe to affected configuration changes
2021-12-09T20:06:28.145Z [jfrt ] [INFO ] [ ] [ffectedConfigStreamObserver:32] [Stream_1639080153144] - publishing full invalidation and attempting to resubscribe to affected configuration changes
2021-12-09T20:06:29.646Z [jfrt ] [INFO ] [ ] [ectedEntitiesStreamObserver:35] [Stream_1639080154645] - publishing full invalidation and attempting to resubscribe to affected permissions changes
2021-12-09T20:07:26.615Z [jfrt ] [ERROR] [59398681a5a1be14] [o.j.a.c.h.AccessHttpClient:136] [http-nio-8081-exec-6] - Error while executing /api/v1/users/ on access. Exception message: Read timed out
2021-12-09T20:07:51.686Z [jfrt ] [ERROR] [6a934e4bf3e126b4] [o.j.a.c.h.AccessHttpClient:136] [http-nio-8081-exec-4] - Error while executing /api/v1/users/ on access. Exception message: Read timed out
2021-12-09T20:07:56.622Z [jfrt ] [ERROR] [2fe4c5688866ed57] [o.j.a.c.h.AccessHttpClient:136] [http-nio-8081-exec-8] - Error while executing /api/v1/users/ on access. Exception message: Read timed out
2021-12-09T20:07:56.622Z [jfrt ] [ERROR] [2b94004eb23bb041] [o.j.a.c.h.AccessHttpClient:136] [ttp-nio-8081-exec-12] - Error while executing /api/v1/users/ on access. Exception message: Read timed out
Terminating Artifactory
This is self-hosted Artifactory 7.27.10 running in Kubernetes
This was a disk performance problem. I use EFS for my Artifactory data volume. For some reason still unknown to me, Artifactory started requiring higher IOPS than the EFS volume could provide. I was able to solve the problem by changing the EFS volume to provisioned IOPS.
Pulls worked fine, because my backing datastore is s3. So the problem only manifested on pushes, after all the docker layers were uploaded, and when log files were written to the EFS volume. It was on log writes that the system choked and restarted.
I'm trying to upgrade my current artifactory 6.23.1 to 7.10.1
All upgrade logs are ok but when I restart the service I've the following errors:
[root ~]# cat /opt/jfrog/artifactory/var/log/console.log | grep ERROR
2020-11-14T14:12:06.384Z [jfrt ] [ERROR] [bca759f3b3ef4148] [o.a.l.v.LoggingVersion:83 ] [ocalhost-startStop-2] - Error occurred while converting logback config for conversion: File '/opt/jfrog/artifactory/var/etc/artifactory/logback.xml' does not exist.
2020-11-14T14:12:06.387Z [jfrt ] [ERROR] [bca759f3b3ef4148] [o.a.l.c.LoggingConverter:69 ] [ocalhost-startStop-2] - Failed to execute logging conversion.
2020-11-14T14:12:20.353Z [jfrt ] [ERROR] [ea891217e8aa6df ] [d.c.m.ConverterManagerImpl:249] [art-init ] - Conversion failed. You should analyze the error and retry launching Artifactory. Error is: The current Artifactory config schema namespace is 'http://artifactory.jfrog.org/xsd/3.1.2' The provided config does not seem to be compliant with it.
2020-11-14T14:12:20.355Z [jfrt ] [ERROR] [ea891217e8aa6df ] [ctoryContextConfigListener:126] [art-init ] - Application could not be initialized: The current Artifactory config schema namespace is 'http://artifactory.jfrog.org/xsd/3.1.2' The provided config does not seem to be compliant with it.
2020-11-14T14:12:22.366Z [jfrt ] [ERROR] [ ] [o.a.w.s.ArtifactoryFilter:213 ] [http-nio-8081-exec-5] - Artifactory failed to initialize: Context is null
2020-11-14T14:15:20.393Z [jffe ] [ERROR] [ ] [ ] [main ] - Error: Error starting application Failed pinging artifactory for 180Request failed with status code 404
2020-11-14T14:15:20.397Z [jffe ] [ERROR] [ ] [ ] [main ] - exit code : 0
And the website is not working (error 500)
What can I do to solve this error ?
Nicolas, it complains about 2 issues,
1.Logback.xml not in the place under $JFROG_HOME/artifactory/var/etc/artfactory location, use the find command to check the location of logback.xml
The Artifactory config complains about incorrect xsd values, can you tell me exactly from which version to which version the Artifactory is upgraded and the exact steps, as well as navigate to $JFROG_HOME/artifactory/var/etc/artfactory and open the artifactory.config.latest.xml and share the first 4 lines.
I upgraded my Artifactory Pro server from version 6.x to 7.x.
I have ldap authentication with MS ActiveDirectory After upgrade some users cannot login to the web-gui with 500 error. I founded error in artifactory logs:
2020-07-17T07:50:44.288Z [jffe ] [ERROR] [6261074d6e4d564 ] [ ] [main ] - http request failed with code - BAD_REQUEST
2020-07-17T07:50:44.288Z [jffe ] [ERROR] [6261074d6e4d564 ] [ ] [main ] - http request failed with message - Could not find user 'ad-test-usr'
2020-07-17T07:50:44.289Z [jffe ] [ERROR] [ ] [ ] [main ] - Error: Request failed with status code 400
at createError (/opt/jfrog/artifactory/app/frontend/bin/server/dist/node_modules/#jfrog/access-nodejs-client/node_modules/axios/lib/core/createError.js:16:15)
at settle (/opt/jfrog/artifactory/app/frontend/bin/server/dist/node_modules/#jfrog/access-nodejs-client/node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (/opt/jfrog/artifactory/app/frontend/bin/server/dist/node_modules/#jfrog/access-nodejs-client/node_modules/axios/lib/adapters/http.js:237:11)
at IncomingMessage.emit (events.js:203:15)
at endReadableNT (_stream_readable.js:1145:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
But user exist in AD and in Artifactory.
It was a bug, in version 6.x logins are case insensitive “UserID” and “userid” are valid both and I can login with any of them.
In version 7.x login is case sensitive and valid is only with lower case “userid”
When executing the Artifactory version 7.4.3, I see below error, and I changed as per the jfrog artifactory could not validate router error but still same error, can someone please advise. Thanks.
failed initializing Access client: Cluster join: Service registry ping failed; context deadline exceeded. last error: Error while trying to connect to local router at address 'http://localhost:8046/access': Get http://localhost:8046/access/api/v1/system/ping: dial tcp 127.0.0.1:8046: connect: connection refused
System.Yaml file:
shared:
## Security Configuration
security:
## Maximum time to wait for key files (master.key and join.key)
#bootstrapKeysReadTimeoutSecs: 120
## Join key value for joining the cluster (takes precedence over 'joinKeyFile')
#joinKey: "<Your joinKey>"
## Join key file location
#joinKeyFile: "<For example: JFROG_HOME/artifactory/var/etc/security/join.key>"
## Master key file location
## Generated by the product on first startup if not provided
#masterKeyFile: "<For example: JFROG_HOME/artifactory/var/etc/security/master.key>"
## Node Settings
node:
## Default: auto resolved by startup script
ip: 172.24.18.44
## Sets this node as primary in HA installation
#primary: true
## A unique id to identify this node.
## Default: auto generated at startup.
#id: "art1"
## Sets this node as part of HA installation
#haEnabled: true
Below is the error from router-service.log, i don't find the shared node ip in router-service.log
[join_executor.go:148 ] [main ] - Cluster join: Retry 120: Service registry ping failed, will retry. Error: Could not parse error from service registry, status code: 404
after updating the system.yaml file:
shared:
node:
ip: 172.24.18.44
router-service.log
2020-05-19T04:36:48.559Z [jfrou] [INFO ] [660aef287984ef11] [bootstrap.go:73 ] [main ] - JFrog Router IP: 172.24.18.44
2020-05-19T04:36:48.562Z [jfrou] [INFO ] [660aef287984ef11] [bootstrap.go:165 ] [main ] - System configuration encryption report:
shared.newrelic.licenseKey: does not exist in the config file
shared.security.joinKeyFile: file '/apps/artifactory-pro-7.4.3/var/etc/security/join.key' - already encrypted
2020-05-19T04:36:48.563Z [jfrou] [INFO ] [660aef287984ef11] [bootstrap.go:78 ] [main ] - JFrog Router Service ID: jfrou#01e8nfmn4nqdr5ay9q2225f6v6
2020-05-19T04:36:48.563Z [jfrou] [INFO ] [660aef287984ef11] [bootstrap.go:79 ] [main ] - JFrog Router Node ID: tsrca24l
2020-05-19T04:37:22.945Z [jfrou] [INFO ] [660aef287984ef11] [join_executor.go:148 ] [main ] - Cluster join: Retry 5: Service registry ping failed, will retry. Error: Get http://localhost:8040/access/api/v1/system/ping: context deadline exceeded
#kumar the system.yaml is not accurate. The node section should be indented under shared as below. You can remove the security section completely.
shared:
node:
ip: <your ip>