Hello everybody when i run:
/opt/open-xchange/sbin/createcontext -A oxadminmaster -P pw_for_master -c 1 -u oxadmin -d "Context Admin" -g Admin -s User -p pw_for_admin -L defaultcontext -e oxadmin#myDomain -q 1024 --access-combination-name=all
following error occurs:
context 1 could not be created:
Server response:
No suitable database available to complete the operation. Please ensure registered databases are reachable and their schemas have enough capacity left and are up-to-date.; exceptionId 321429037-66
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorage.getNextDBHandleByWeight(OXUtilMySQLStorage.java:4080)
at com.openexchange.admin.plugin.hosting.storage.mysqlStorage.OXContextMySQLStorage.create(OXContextMySQLStorage.java:1648)
at com.openexchange.admin.plugin.hosting.rmi.impl.OXContext.createmaincall(OXContext.java:1240)
at com.openexchange.admin.rmi.impl.OXContextCommonImpl.createcommon(OXContextCommonImpl.java:178)
at com.openexchange.admin.plugin.hosting.rmi.impl.OXContext.create(OXContext.java:462)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.create(Unknown Source)
at com.openexchange.admin.console.context.Create.maincall(Create.java:150)
at com.openexchange.admin.console.context.CreateCore.commonfunctions(CreateCore.java:149)
at com.openexchange.admin.console.context.Create.execute(Create.java:94)
at com.openexchange.admin.console.context.Create.main(Create.java:85)
I did run
/opt/open-xchange/sbin/initconfigdb --configdb-pass=db_password -a --mysql-root-passwd=root_password
/opt/open-xchange/sbin/oxinstaller --no-license --servername=oxserver --configdb-user=openexchange --configdb-writehost=localhost --network-listener-host=localhost --servermemory 2048 --master-pass=pw_for_master --configdb-pass=pw_for_db --network-listener-host=localhost --ajp-bind-port=localhost
And could register:
database (groupware database)
server
filestore;
but i am not quite sure, what i registered as database, because with show databases;
i do not find my specified database (name is oxdatabase) in the results.
+--------------------+
| Database |
+--------------------+
| configdb |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.001 sec)
When i created oxdatabase manually and ran
/opt/open-xchange/sbin/createschemas -A oxadminmaster -P pw_for_master -n oxdatabase
but this resulted in:
java.sql.SQLException: Access denied for user 'openexchange'#'localhost' (using password: YES); exceptionId 321429037-68
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorageCommon.getSimpleSQLConnectionFor(OXUtilMySQLStorageCommon.java:137)
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorageCommon.listDatabases(OXUtilMySQLStorageCommon.java:156)
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorage.createDatabaseSchemas(OXUtilMySQLStorage.java:1389)
at com.openexchange.admin.rmi.impl.OXUtil.createSchemas(OXUtil.java:475)
at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.createSchemas(Unknown Source)
at com.openexchange.admin.console.util.database.CreateSchemas$1.run(CreateSchemas.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
I checked the settings: PW is ok; user is ok; grants should be ok (i granted star.star);
On disk about 90 GB left; only one partition.
OS: Debian 10
I run as root
So my questions are:
Does my error occur because of my (missing) database?
If yes could someone provide me with the schema for the groupware database?
And why does the error in createschemas occur?
I could fix it.
First step - maybe optional
Re-install Buster
Secondly change hostname to server name -> e.g oxserver
Related
i have problem with cant force restart from management console login as admin. can anyone helping with me about this. i got log like this
`[2022-10-01 00:45:04,581] INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2022-10-01 00:45:04,581] INFO - CarbonCoreActivator Operating System : Linux 4.18.0-305.el8.x86_64, amd64
[2022-10-01 00:45:04,581] INFO - CarbonCoreActivator Java Home : /usr/java/jdk1.8.0_341-amd64/jre
[2022-10-01 00:45:04,581] INFO - CarbonCoreActivator Java Version : 1.8.0_341
[2022-10-01 00:45:04,581] INFO - CarbonCoreActivator Java VM : Java HotSpot(TM) 64-Bit Server VM 25.341-b10,Oracle Corporation
[2022-10-01 00:45:04,582] INFO - CarbonCoreActivator Carbon Home : /opt/source/wso2_binary/wso2am-4.1.0
[2022-10-01 00:45:04,582] INFO - CarbonCoreActivator Java Temp Dir : /opt/source/wso2_binary/wso2am-4.1.0/tmp
[2022-10-01 00:45:04,582] INFO - CarbonCoreActivator User : abhimata, en-US, Asia/Jakarta
[2022-10-01 00:45:04,786] INFO - DefaultCryptoProviderComponent 'CryptoService.Secret' property has not been set. 'org.wso2.carbon.crypto.provider.SymmetricKeyInternalCryptoProvider' won't be registered as an internal crypto provider. Please set the secret if the provider needs to be registered.
[2022-10-01 00:45:05,125] INFO - KafkaEventAdapterServiceDS Successfully deployed the Kafka output event adaptor service
[2022-10-01 00:45:05,279] INFO - TemplateDeployerServiceTrackerDS Successfully deployed the execution manager tracker service
[2022-10-01 00:45:06,716] INFO - ServiceComponent Eventing Hub ServiceComponent is activated
[2022-10-01 00:45:07,431] WARN - Digester Match [Server/Service/Engine/Host/Valve] failed to set property [maxDays] to []
[2022-10-01 00:45:08,095] ERROR - DefaultRealm nullType class java.lang.reflect.InvocationTargetException
org.wso2.carbon.user.core.UserStoreException: nullType class java.lang.reflect.InvocationTargetException
at org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:404) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.common.DefaultRealm.initializeObjects(DefaultRealm.java:231) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:136) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:276) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:102) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:115) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:72) ~[?:?]
at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61) ~[?:?]
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:842) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_341]
at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:834) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:791) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1013) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.Module.doStart(Module.java:598) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.Module.start(Module.java:462) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$1.run(ModuleContainer.java:1820) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$2$1.execute(EquinoxContainerAdaptor.java:150) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1813) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1770) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1735) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1661) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345) ~[org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_341]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_341]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_341]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_341]
at org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:358) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
... 25 more
Caused by: org.wso2.carbon.user.core.UserStoreException: DB error occurred while persisting domain : PRIMARY & tenant id : -1234
at org.wso2.carbon.user.core.util.UserCoreUtil.persistDomain(UserCoreUtil.java:931) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.persistDomain(AbstractUserStoreManager.java:9083) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager.<init>(JDBCUserStoreManager.java:320) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.jdbc.JDBCUserStoreManager.<init>(JDBCUserStoreManager.java:262) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at org.wso2.carbon.user.core.jdbc.UniqueIDJDBCUserStoreManager.<init>(UniqueIDJDBCUserStoreManager.java:129) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_341]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_341]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_341]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_341]
at org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:358) ~[org.wso2.carbon.user.core_4.6.3.jar:?]
... 25 more
i expecting can force restart from management console or can with restart with command line from linux. please help me about this; thanks
Usually you can stop wso2server gracefully with ./wso2server.sh --stop. (./api-manager.sh --stop for latest versions)
In the management console(https://localhost:9443/carbon) also there is a shutdown button that you can use.
If you need to kill the process as you require right now, you can use in inbuilt Linux commands such as ps to locate the process and kill it.
Enter the command ps ax | grep wso2
This will list the set of processes started that has the text wso2. Unless you opened several apim instances, there will be one process that contains an ID. There, the first number refers to the process id.
Enter following command to gracefully shutdown the APIM.
kill <process-id>
You may force the shutdown by sending signal kill (-9) like follows if you need to.
kill -9 <process-id>
With ps again, if you can see there is a java process that sets the java home to <APIM_HOME>, additionally, you can kill it with
kill -9 java
I have upgraded from Artifactoy 5.1 to 5.4. Now Artifactory will now start. Getting the following error:
INFO: Starting ProtocolHandler ["ajp-nio-8019"]
2017-06-22 09:59:34,971 [art-init] [INFO ] (o.a.s.a.AccessServiceImpl:243) - Got response from Access server after 2687 ms, continuing.
2017-06-22 09:59:35,387 [art-init] [ERROR] (o.a.w.s.ArtifactoryContextConfigListener:97) - Application could not be initialized: HTTP response status 401:{"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]}
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_91]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_91]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_91]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_91]
at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener.configure(ArtifactoryContextConfigListener.java:222) ~[artifactory-web-application-5.4.0.jar:na]
at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener.access$2(ArtifactoryContextConfigListener.java:184) ~[artifactory-web-application-5.4.0.jar:na]
at org.artifactory.webapp.servlet.ArtifactoryContextConfigListener$1.run(ArtifactoryContextConfigListener.java:93) ~[artifactory-web-application-5.4.0.jar:na]
Caused by: org.springframework.beans.factory.BeanInitializationException: Failed to initialize bean 'org.artifactory.security.access.AccessService'.; nested exception is java.lang.RuntimeException: Failed to generate service admin token using bootstrap credentials.
at org.artifactory.spring.ArtifactoryApplicationContext.refresh(ArtifactoryApplicationContext.java:230) ~[artifactory-core-5.4.0.jar:na]
at org.artifactory.spring.ArtifactoryApplicationContext.<init>(ArtifactoryApplicationContext.java:114) ~[artifactory-core-5.4.0.jar:na]
... 7 common frames omitted
Caused by: java.lang.RuntimeException: Failed to generate service admin token using bootstrap credentials.
at org.jfrog.access.client.AccessClientBootstrap.createAndStoreServiceAdminToken(AccessClientBootstrap.java:110) ~[access-client-core-2.0.0.jar:na]
at org.jfrog.access.client.AccessClientBootstrap.bootstrapServiceAdminToken(AccessClientBootstrap.java:79) ~[access-client-core-2.0.0.jar:na]
at org.jfrog.access.client.AccessClientBootstrap.<init>(AccessClientBootstrap.java:42) ~[access-client-core-2.0.0.jar:na]
at org.artifactory.security.access.AccessServiceImpl.initAccessService(AccessServiceImpl.java:227) ~[artifactory-core-5.4.0.jar:na]
at org.artifactory.security.access.AccessServiceImpl.initIfNeeded(AccessServiceImpl.java:216) ~[artifactory-core-5.4.0.jar:na]
at org.artifactory.security.access.AccessServiceImpl.init(AccessServiceImpl.java:211) ~[artifactory-core-5.4.0.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_91]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_91]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at org.artifactory.storage.fs.lock.aop.LockingAdvice.invoke(LockingAdvice.java:76) ~[artifactory-storage-common-5.4.0.jar:na]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
at com.sun.proxy.$Proxy144.init(Unknown Source) ~[na:na]
at org.artifactory.spring.ArtifactoryApplicationContext.refresh(ArtifactoryApplicationContext.java:228) ~[artifactory-core-5.4.0.jar:na]
... 8 common frames omitted
Caused by: org.jfrog.access.client.AccessClientHttpException: HTTP response status 401:{"errors":[{"code":"UNAUTHORIZED","detail":"Bad credentials","message":"HTTP 401 Unauthorized"}]}
at org.jfrog.access.client.http.AccessHttpClient.createRestResponse(AccessHttpClient.java:312) ~[access-client-core-2.0.0.jar:na]
at org.jfrog.access.client.http.AccessHttpClient.restCall(AccessHttpClient.java:299) ~[access-client-core-2.0.0.jar:na]
at org.jfrog.access.client.http.AccessHttpClient.createToken(AccessHttpClient.java:133) ~[access-client-core-2.0.0.jar:na]
at org.jfrog.access.client.token.TokenClientImpl.create(TokenClientImpl.java:36) ~[access-client-core-2.0.0.jar:na]
at org.jfrog.access.client.AccessClientBootstrap.createAndStoreServiceAdminToken(AccessClientBootstrap.java:103) ~[access-client-core-2.0.0.jar:na]
... 28 common frames omitted
2017-06-22 09:59:41,768 [http-nio-8081-exec-7] [ERROR] (o.a.w.s.ArtifactoryFilter:188) - Artifactory failed to initialize: Context is null
Jun 22, 2017 10:04:14 AM org.apache.catalina.core.StandardServer await
This error occurs due to a missing step during the upgrade process. As mentioned in the wiki page, as part of the upgrade, you need to remove the existing $ARTIFACTORY_HOME/bin folder and copy over the new one from the extracted zip file.
This step is crucial, as it contains a property which makes the Access service to be bundled with Artifactory. When this property is missing, the Access service creates a new database with a new admin token, which is different than the existing one. This results in the 401 error that you're seeing, which prevents Artifactory from being restarted.
In order to overcome this issue, follow the steps of the upgrade process, including the removal of the existing bin folder.
Command:
wcbdd#ubuntu:~/apache/oozie-4.1.0/distro/target/oozie-4.1.0/bin$ sudo -u oozie ./ooziedb.sh create -sqlfile oozie.sql –run
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Validate DB Connection
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils
at org.apache.oozie.service.Services.setServiceInternal(Services.java:374)
at org.apache.oozie.service.Services.<init>(Services.java:110)
at org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:163)
at org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:845)
at org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:853)
at org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:181)
at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:125)
at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:76)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.ReflectionUtils
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 8 more
I am using Karaf 2.3.0 for deploying my OSGi activator bundles and exposing my remote services as rest enabled. Thigs are working fine. Only once I got an out of memory error on the karaf logs (attaching the logs), after which I was not able to access my rest serviced. When I took the heap and thread dump after an hour of server crash (my karaf process is still running even after the OOMemory error), I could find nothing big in the dumps. Checked the process using jvisualvm (karaf process Xmx-1024 MB) and the memory and cpu consumptions are the bare minimal. But I am not bot able to access any of my services. Trying to access the services keeps trying for minutes without any error until timeout. I do not see the server log printing any signs of my access as well.. Is'nt the process supposed to stop once the out of mem error occurs? How can i identify what could caused the problem? Attaching the logs below
2014-07-16 12:19:51,461 | WARN | qtp1863802945-52 | ServletHandler | lipse.jetty.util.log.JavaUtilLog 70 | 52 - org.eclipse.jetty.util - 7.6.7.v20120910 | /services/alertThreshold/F5/VirtualServer
java.lang.reflect.UndeclaredThrowableException
at org.ops4j.pax.web.service.internal.$Proxy11.service(Unknown Source)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:447)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[59:org.eclipse.jetty.security:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:74)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.Server.handle(Server.java:363)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at org.ops4j.pax.web.service.internal.HttpServiceStarted$2.invoke(HttpServiceStarted.java:210)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
... 27 more
Caused by: javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[93:com.sun.jersey.servlet:1.15.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)[51:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.1.2]
... 32 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2014-07-16 12:19:51,460 | WARN | 2.168.31.36:5000 | cluster | verMonitor$ServerMonitorRunnable 117 | 115 - org.db.mongo - 1.0.0.201407140926 | Exception in monitor thread during notification of server state change
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:2367)[:1.7.0_17]
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)[:1.7.0_17]
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)[:1.7.0_17]
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)[:1.7.0_17]
at java.lang.StringBuilder.append(StringBuilder.java:132)[:1.7.0_17]
at com.mongodb.ServerDescription.getShortDescription(ServerDescription.java:467)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.ClusterDescription.getShortDescription(ClusterDescription.java:191)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.BaseCluster.updateDescription(BaseCluster.java:158)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster.updateDescription(MultiServerCluster.java:240)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster.onChange(MultiServerCluster.java:149)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster.access$100(MultiServerCluster.java:40)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster$DefaultServerStateListener.stateChanged(MultiServerCluster.java:111)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.DefaultServer$DefaultServerStateListener.stateChanged(DefaultServer.java:104)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.ServerMonitor$ServerMonitorRunnable.run(ServerMonitor.java:114)[115:org.db.mongo:1.0.0.201407140926]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
2014-07-16 12:19:51,458 | ERROR | qtp1863802945-55 | ContainerResponse | .spi.container.ContainerResponse 406 | 92 - com.sun.jersey.jersey-server - 1.15.0 | The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: Java heap space
at sun.nio.cs.UTF_8.newDecoder(UTF_8.java:68)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:131)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:122)[:1.7.0_17]
at java.lang.StringCoding.decode(StringCoding.java:187)[:1.7.0_17]
at java.lang.String.<init>(String.java:416)[:1.7.0_17]
at org.bson.BasicBSONDecoder$BSONInput.readUTF8String(BasicBSONDecoder.java:544)
at org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:230)
at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:154)
at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:132)
at com.mongodb.DefaultDBDecoder.decode(DefaultDBDecoder.java:62)
at com.mongodb.Response.<init>(Response.java:85)
at com.mongodb.DBPort$1.execute(DBPort.java:141)
at com.mongodb.DBPort$1.execute(DBPort.java:135)
at com.mongodb.DBPort.doOperation(DBPort.java:164)
at com.mongodb.DBPort.call(DBPort.java:135)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:292)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:271)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:237)
at com.mongodb.QueryResultIterator.getMore(QueryResultIterator.java:137)
at com.mongodb.QueryResultIterator.hasNext(QueryResultIterator.java:127)
at com.mongodb.DBCursor._hasNext(DBCursor.java:551)
at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
at com.testing.apptest.search.dao.ObjectSearchDao.getObjects(ObjectSearchDao.java:1648)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectList(ObjectDictionaryManagement.java:572)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectDict(ObjectDictionaryManagement.java:358)
at com.testing.apptest.core.service.alertSettings.ThresholdSettingsManagement.getObjectList(ThresholdSettingsManagement.java:239)
at com.testing.apptest.rest.ThresholdSettingsRest.getDeviceList(ThresholdSettingsRest.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[92:com.sun.jersey.jersey-server:1.15.0]
2014-07-16 12:19:51,457 | ERROR | qtp1863802945-57 | ContainerResponse | .spi.container.ContainerResponse 406 | 92 - com.sun.jersey.jersey-server - 1.15.0 | The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-07-16 12:19:52,669 | WARN | qtp1863802945-57 | ServletHandler | lipse.jetty.util.log.JavaUtilLog 70 | 52 - org.eclipse.jetty.util - 7.6.7.v20120910 | /services/notificationCenter
java.lang.reflect.UndeclaredThrowableException
at org.ops4j.pax.web.service.internal.$Proxy11.service(Unknown Source)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:447)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[59:org.eclipse.jetty.security:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:74)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.Server.handle(Server.java:363)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at org.ops4j.pax.web.service.internal.HttpServiceStarted$2.invoke(HttpServiceStarted.java:210)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
... 27 more
Caused by: javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[93:com.sun.jersey.servlet:1.15.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)[51:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.1.2]
... 32 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-07-16 12:19:52,669 | WARN | qtp1863802945-55 | ServletHandler | lipse.jetty.util.log.JavaUtilLog 70 | 52 - org.eclipse.jetty.util - 7.6.7.v20120910 | /services/alertThreshold/F5/WideIp
java.lang.reflect.UndeclaredThrowableException
at org.ops4j.pax.web.service.internal.$Proxy11.service(Unknown Source)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:447)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[59:org.eclipse.jetty.security:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:74)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.Server.handle(Server.java:363)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at org.ops4j.pax.web.service.internal.HttpServiceStarted$2.invoke(HttpServiceStarted.java:210)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
... 27 more
Caused by: javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[93:com.sun.jersey.servlet:1.15.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)[51:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.1.2]
... 32 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at sun.nio.cs.UTF_8.newDecoder(UTF_8.java:68)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:131)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:122)[:1.7.0_17]
at java.lang.StringCoding.decode(StringCoding.java:187)[:1.7.0_17]
at java.lang.String.<init>(String.java:416)[:1.7.0_17]
at org.bson.BasicBSONDecoder$BSONInput.readUTF8String(BasicBSONDecoder.java:544)
at org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:230)
at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:154)
at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:132)
at com.mongodb.DefaultDBDecoder.decode(DefaultDBDecoder.java:62)
at com.mongodb.Response.<init>(Response.java:85)
at com.mongodb.DBPort$1.execute(DBPort.java:141)
at com.mongodb.DBPort$1.execute(DBPort.java:135)
at com.mongodb.DBPort.doOperation(DBPort.java:164)
at com.mongodb.DBPort.call(DBPort.java:135)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:292)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:271)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:237)
at com.mongodb.QueryResultIterator.getMore(QueryResultIterator.java:137)
at com.mongodb.QueryResultIterator.hasNext(QueryResultIterator.java:127)
at com.mongodb.DBCursor._hasNext(DBCursor.java:551)
at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
at com.testing.apptest.search.dao.ObjectSearchDao.getObjects(ObjectSearchDao.java:1648)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectList(ObjectDictionaryManagement.java:572)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectDict(ObjectDictionaryManagement.java:358)
at com.testing.apptest.core.service.alertSettings.ThresholdSettingsManagement.getObjectList(ThresholdSettingsManagement.java:239)
at com.testing.apptest.rest.ThresholdSettingsRest.getDeviceList(ThresholdSettingsRest.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[92:com.sun.jersey.jersey-server:1.15.0]
First of all a process isn't supposed to stop after a OOMError. An OutOfMemoryError is a Throwable and therefore can be handled by the virtual machine. In your case the bundle in question did get removed and therefore the memory consumption is ok again. You'll need to restart your application Bundle and maybe the Pax Web one too.
Regarding your exception I've seen that there is an issue with an ArrayCopy, you seem to have a lot of "big" objects in your application, maybe just increasing the Heap-Size is already enough.
You can set the Xmx (JAVA_MAX_MEM and JAVA_MAX_PERM_MEM) in $KARAF_HOME/bin/setenv
I setup 2 nodes (virtaul machines) hadoop cluster setup. After successfully starting dfs and mapred deamons , I run hadoop demo examples, Program gets slow after this terminal display:
Number of Maps = 4 Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Starting Job`enter code here`
13/06/10 21:36:43 INFO mapred.FileInputFormat: Total input paths to process : 4
13/06/10 21:36:43 INFO mapred.FileInputFormat: Total input paths to process : 4
13/06/10 21:36:43 INFO mapred.JobClient: Running job: job_201306101254_0005
13/06/10 21:36:44 INFO mapred.JobClient: map 0% reduce 0%
13/06/10 21:36:49 INFO mapred.JobClient: map 75% reduce 0%
13/06/10 21:36:50 INFO mapred.JobClient: map 100% reduce 0%
So basically map tasks are being completed correctly.I checked this behaviour after looking at attempt log, particaularly attempt log of reduce task which confirm that the reduce task is unable to read mapoutput generated by the other slave**, error is like this:
****13/06/11 01:55:45 WARN mapred.JobClient: Error reading task outputhttp://hadoop-desk.localdomain:50060/tasklog?plaintext=true&taskid=attempt_201306110154_0001_m_000000_0&filter=stdout
13/06/11 01:55:45 WARN mapred.JobClient: Error reading task outputhttp://hadoop-desk.localdomain:50060/tasklog?plaintext=true&taskid=attempt_201306110154_0001_m_000000_0&filter=stderr
13/06/11 01:55:49 INFO mapred.JobClient: map 75% reduce 16%****
So the map task which generated this mapoutput is considered fail and rescheduled on different slave(slave in which reduce is running), making whole program slow. I think the reason is because of the etc/hosts of ubuntu which is:
127.0.0.1 localhost
127.0.1.1 hadoop-desk.localdomain hadoop-desk
192.168.196.128 master
192.168.196.129 slave
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
I am getting the same error even after removing localhost,this line
127.0.0.1 localhost
And I removed this line
127.0.1.1 hadoop-desk.localdomain hadoop-desk
Then i got this error:
Number of Maps = 4 Samples per Map = 10000
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs.SafeModeException: Cannot delete /user/hadoop-user/test-mini-mr. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.
at org.apache.hadoop.dfs.FSNamesystem.deleteInternal(FSNamesystem.java:1494)
at org.apache.hadoop.dfs.FSNamesystem.delete(FSNamesystem.java:1466)
at org.apache.hadoop.dfs.NameNode.delete(NameNode.java:425)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
at org.apache.hadoop.ipc.Client.call(Client.java:715)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
at org.apache.hadoop.dfs.$Proxy0.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy0.delete(Unknown Source)
at org.apache.hadoop.dfs.DFSClient.delete(DFSClient.java:529)
at org.apache.hadoop.dfs.DistributedFileSystem.delete(DistributedFileSystem.java:192)
at org.apache.hadoop.examples.PiEstimator.launch(PiEstimator.java:188)
at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:245)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:252)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
ubuntu 8 (dont know about above versions) has this entry in etc/hosts file
127.0.1.1 yourhostname.localdomain yourhostname
this entry was creating problems so comment that line, Heres my "etc/hosts" file:
127.0.0.1 localhost
#127.0.1.1 hadoop-desk.localdomain hadoop-desk
192.168.196.128 master
192.168.196.129 slave
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
after that this error will came:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.dfs.SafeModeException: <Cannot delete /user/hadoop-user/test-mini-mr>. Name node is in safe mode.
To avoid this error change this dfs.safemode.threshold.pct parameter in hdfs-default.xml from 0.999f to 0.0f
<property>
<name>dfs.safemode.threshold.pct</name>
<value>0.00f</value>
<description>
Specifies the percentage of blocks that should satisfy
the minimal replication requirement defined by dfs.replication.min.
Values less than or equal to 0 mean not to wait for any particular
percentage of blocks before exiting safemode.
Values greater than 1 will make safe mode permanent.
</description>
</property>
Now I was still getting "error reading mapoutput" (mapoutput of different slave). I had same hostname for both master and slave, so I changed their hostname to "master" and "slave" respectively by editing their respective "etc/hostname" files. Since I already had entry for "master" and "slave" in etc/hosts file, I didnt get any error.