Artifactory JFrog Backup failing with error code 401 - artifactory

Question

Will appreciate if someone can please guide or provide me pointers to debug the backup issues with Artifactory. Whenever backups are performed - there is always an error message of 401 on /api/v1/system/backup/export in the artifactory.log
The backups exist on backup location but with an error message in the log. Not sure how can debug and implications of this error in the logs. I can see in the stack the rest call is failing, have tried setting a password to unsupported and multiple other things but the error persists. Also checked the Jira on Artifactory to no avail. Any pointers will be greatly appreciated
More details
artifactory.version=5.9.3
artifactory.timestamp=1521564024289
artifactory.revision=50903900
artifactory.buildNumber=820
The backups are failing with following the info in the log.
2018-04-24 11:59:24,620 [ajp-apr-8009-exec-9] [ERROR] (o.a.s.a.AccessServiceImpl:1070) - Error during access server backup
org.jfrog.access.client.AccessClientHttpException: HTTP response status 401:Failed on executing /api/v1/system/backup/export, with response: {"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]}
at org.jfrog.access.client.http.AccessHttpClient.createRestResponse(AccessHttpClient.java:154)
at org.jfrog.access.client.http.AccessHttpClient.restCall(AccessHttpClient.java:113)
...
Following is the full stack as displayed in artifactory.log
2018-04-24 11:59:24,620 [ajp-apr-8009-exec-9] [ERROR] (o.a.s.a.AccessServiceImpl:1070) - Error during access server backup
org.jfrog.access.client.AccessClientHttpException: HTTP response status 401:Failed on executing /api/v1/system/backup/export, with response: {"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]}
at org.jfrog.access.client.http.AccessHttpClient.createRestResponse(AccessHttpClient.java:154)
at org.jfrog.access.client.http.AccessHttpClient.restCall(AccessHttpClient.java:113)
at org.jfrog.access.client.system.SystemClientImpl.exportAccessServer(SystemClientImpl.java:21)
at org.artifactory.security.access.AccessServiceImpl.exportTo(AccessServiceImpl.java:1060)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
at com.sun.proxy.$Proxy144.exportTo(Unknown Source)
at org.artifactory.spring.ArtifactoryApplicationContext.exportTo(ArtifactoryApplicationContext.java:662)
at org.artifactory.ui.rest.service.admin.importexport.exportdata.ExportSystemService.execute(ExportSystemService.java:67)
at org.artifactory.rest.common.service.ServiceExecutor.process(ServiceExecutor.java:38)
at org.artifactory.rest.common.resource.BaseResource.runService(BaseResource.java:92)
at org.artifactory.ui.rest.resource.admin.importexport.ExportArtifactResource.exportSystem(ExportArtifactResource.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:184)
at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:93)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:403)
at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:212)
at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:166)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:128)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
at org.apache.coyote.ajp.AbstractAjpProcessor.process(AbstractAjpProcessor.java:877)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2527)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2516)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
2018-04-24 11:59:24,660 [ajp-apr-8009-exec-9] [INFO ] (o.a.s.ArtifactoryApplicationContext:819) - Note: the etc exported folder has excessive permissions. Be careful with the files.

Answer 1

Thanks for the feedback.
I suspect the reason for the behavior is that for the more common operations Artifactory uses the <adminToken> from the Config Descriptor, however some other operations use credentials saved in $ARTIFACTORY_HOME/etc/security/access/access.creds.
These seem to be bad.
In order to restart these you should:
Create the following file: $ARTIFACTORY_HOME/access/etc/bootstrap.creds
The content of the file should be: access-admin#*=password
The permissions of the file must be 600.
Once that file is in place, restart Artifactory.

Answer 2

Since Artifactory version 5.4 there is a separate service in Artifactory named "Access", which manages permissions and security related configurations.
Artifactory authenticates with Access using an access token that is configured in the "Config Descriptor".
It is possible that this issue will be resolved by a simple restart, so that's what i would have started with.
If this doesn't work, or what i suspect will happen is Artifactory won't be able to start, please do (while Artifactory is shut down):
Remove the <adminToken> tag from the $ARTIFACTORY_HOME/etc/artifactory.config.latest.xml
$ cp artifactory.config.latest.xml artifactory.config.import.xml
Restart Artifactory.

Answer 3

Added bootstrap.creds
tomcat#XXXXXXXX:/home/bitnami/apps/artifactory/artifactory_home/etc$ ls -ltrh
total 136K
-rw-r--r-- 1 tomcat tomcat 5.5K Mar 20 16:41 mimetypes.xml
-rw-r--r-- 1 tomcat tomcat 12K Mar 20 16:41 logback.xml
-rw-r--r-- 1 tomcat tomcat 13K Mar 20 16:41 artifactory.system.properties
-rw-r--r-- 1 tomcat tomcat 11K Mar 20 16:41 artifactory.config.bootstrap.xml
-rw-rw-r-- 1 tomcat tomcat 145 Mar 21 13:36 db.properties
drwxrwxr-x 2 tomcat tomcat 4.0K Mar 21 13:37 ui
drwxrwxr-x 2 tomcat tomcat 4.0K Mar 21 13:37 plugins
drwx------ 3 tomcat tomcat 4.0K Mar 21 13:37 security
-rw-rw-r-- 1 tomcat tomcat 13K Mar 21 13:37
artifactory.config.latest.1521639475000.xml
-rw-r--r-- 1 tomcat tomcat 199 Apr 24 18:50 binarystore.xml
-rw-rw-r-- 1 tomcat tomcat 13K Apr 24 18:57 artifactory.config.latest.xml_bkp
-rw-rw-r-- 1 tomcat tomcat 12K Apr 24 18:58
artifactory.config.latest.1524596291000.xml
-rw-rw-r-- 1 tomcat tomcat 14K Apr 25 14:39 artifactory.config.latest.xml
-rw------- 1 tomcat tomcat 24 Apr 25 15:09 bootstrap.creds
-rw-rw-r-- 1 tomcat tomcat 864 Apr 25 15:12 artifactory.properties
Contents of bootstrap.creds are
access-admin#*=password
AND restarted the arttifactory BUT still seeing the same error message
2018-04-25 15:14:29,607 [art-exec-1] [ERROR] (o.a.s.a.AccessServiceImpl:1070) - Error during access server backup: HTTP response status 401:Failed on executing /api/v1/system/backup/export, with response: {"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]}
I tried the backup using access-admin as well as user from gui with same error (both are admin users).
Following are from the access log when artifactory restarted
2018-04-25 15:10:54,396 [localhost-startStop-1] [INFO ] (o.j.a.c.AccessApplicationContextInitializer:46) - Using JFrog Access home at '/opt/bitnami/apps/artifactory/artifactory_home/access'
2018-04-25 15:10:54,413 [localhost-startStop-1] [INFO ] (o.j.a.AccessApplication:48) - Starting AccessApplication v3.2.2 on ip-10-51-35-238 with PID 16169 (/opt/bitnami/apache-tomcat/webapps/access/WEB-INF/lib/access-application-3.2.2.jar started by tomcat in /opt/bitnami/apache-tomcat/bin)
2018-04-25 15:10:54,414 [localhost-startStop-1] [INFO ] (o.j.a.AccessApplication:597) - The following profiles are active: production,grpc
2018-04-25 15:11:11,402 [localhost-startStop-1] [INFO ] (o.j.a.s.d.u.AccessJdbcHelperImpl:129) - Database: Apache Derby 10.11.1.1 - (1616546). Driver: Apache Derby Embedded JDBC Driver 10.11.1.1 - (1616546)
2018-04-25 15:11:11,410 [localhost-startStop-1] [INFO ] (o.j.a.s.d.u.AccessJdbcHelperImpl:132) - Connection URL: jdbc:derby:/opt/bitnami/apps/artifactory/artifactory_home/data/derby
2018-04-25 15:11:16,127 [localhost-startStop-1] [INFO ] (o.j.a.s.s.c.InternalConfigurationServiceImpl:94) - Loading configuration from db finished successfully
2018-04-25 15:11:18,035 [localhost-startStop-1] [INFO ] (o.j.a.s.s.AccessStartupServiceImpl:79) - Found master.key file at /opt/bitnami/apps/artifactory/artifactory_home/etc/security/master.key, using it as master.key
2018-04-25 15:11:19,486 [localhost-startStop-1] [INFO ] (o.j.a.s.s.t.TokenServiceImpl:100) - Scheduling task for revoking expired tokens using cron expression: 0 0 0/1 * * ?
2018-04-25 15:11:19,651 [localhost-startStop-1] [INFO ] (o.j.a.s.r.c.RpcServiceInvoker:86) - Added service: sync
2018-04-25 15:11:20,764 [localhost-startStop-1] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:93) - [ACCESS BOOTSTRAP] Starting JFrog Access bootstrap...
2018-04-25 15:11:23,381 [localhost-startStop-1] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:146) - [ACCESS BOOTSTRAP] Updating server '509b68e8-48e5-4bba-8ea7-6564c65d5b37' private key finger print to: 63c2f42824ead3169fc13eab66d3d254d25c659ceef5cdeed21c3110f47ee0d3
2018-04-25 15:11:24,164 [localhost-startStop-1] [INFO ] (o.j.a.s.AccessServerBootstrapImpl:108) - [ACCESS BOOTSTRAP] JFrog Access bootstrap finished.
2018-04-25 15:11:30,624 [localhost-startStop-1] [INFO ] (o.j.a.s.s.s.RefreshableScheduledJob:56) - Scheduling heartbeat task to run every 5 seconds
2018-04-25 15:11:43,339 [localhost-startStop-1] [INFO ] (o.j.a.AccessApplication:57) - Started AccessApplication in 53.93 seconds (JVM running for 93.388)
2018-04-25 15:11:47,756 [localhost-startStop-1] [WARN ] (o.a.t.u.s.StandardJarScanner:311) - Failed to scan [file:/opt/bitnami/apache-tomcat/lib/derbyLocale_cs.jar] from classloader hierarchy enter code here
Additionaly I enabled trace on the logs. Following is what I am seeing in
/home/bitnami/apps/artifactory/artifactory_home/access/logs/request.log. Seems the ones with backup call are tagged as coming from anonymous vs |jfrt#01c94cf2cpdmx60xvtsf8j1jy4 for other calls.
2018-04-25T15:49:03.906+0000|127.0.0.1|jfrt#01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/groups/|200|0|198|JFrog Access Java Client/3.2.2
2018-04-25T15:50:17.553+0000|127.0.0.1|anonymous|POST|http://127.0.0.1/access/api/v1/system/backup/export|401|0|188|JFrog Access Java Client/3.2.2
2018-04-25T15:50:18.657+0000|127.0.0.1|jfrt#01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/users/|200|0|84|JFrog Access Java Client/3.2.2
2018-04-25T15:50:18.733+0000|127.0.0.1|jfrt#01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/groups/|200|0|73|JFrog Access Java Client/3.2.2
2018-04-25T15:50:42.823+0000|127.0.0.1|anonymous|POST|http://127.0.0.1/access/api/v1/system/backup/export|401|0|134|JFrog Access Java Client/3.2.2
2018-04-25T15:50:42.995+0000|127.0.0.1|jfrt#01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/users/|200|0|114|JFrog Access Java Client/3.2.2
2018-04-25T15:50:43.086+0000|127.0.0.1|jfrt#01c94cf2cpdmx60xvtsf8j1jy4|GET|http://127.0.0.1/access/api/v1/groups/|200|0|64|JFrog Access Java Client/3.2.2
Please let me know if I can provide additional relevant information that can help you to get more insight in my envoirnment. Thank you again for you help