Karaf out of memory error - out-of-memory

I am using Karaf 2.3.0 for deploying my OSGi activator bundles and exposing my remote services as rest enabled. Thigs are working fine. Only once I got an out of memory error on the karaf logs (attaching the logs), after which I was not able to access my rest serviced. When I took the heap and thread dump after an hour of server crash (my karaf process is still running even after the OOMemory error), I could find nothing big in the dumps. Checked the process using jvisualvm (karaf process Xmx-1024 MB) and the memory and cpu consumptions are the bare minimal. But I am not bot able to access any of my services. Trying to access the services keeps trying for minutes without any error until timeout. I do not see the server log printing any signs of my access as well.. Is'nt the process supposed to stop once the out of mem error occurs? How can i identify what could caused the problem? Attaching the logs below
2014-07-16 12:19:51,461 | WARN | qtp1863802945-52 | ServletHandler | lipse.jetty.util.log.JavaUtilLog 70 | 52 - org.eclipse.jetty.util - 7.6.7.v20120910 | /services/alertThreshold/F5/VirtualServer
java.lang.reflect.UndeclaredThrowableException
at org.ops4j.pax.web.service.internal.$Proxy11.service(Unknown Source)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:447)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[59:org.eclipse.jetty.security:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:74)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.Server.handle(Server.java:363)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at org.ops4j.pax.web.service.internal.HttpServiceStarted$2.invoke(HttpServiceStarted.java:210)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
... 27 more
Caused by: javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[93:com.sun.jersey.servlet:1.15.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)[51:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.1.2]
... 32 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2014-07-16 12:19:51,460 | WARN | 2.168.31.36:5000 | cluster | verMonitor$ServerMonitorRunnable 117 | 115 - org.db.mongo - 1.0.0.201407140926 | Exception in monitor thread during notification of server state change
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:2367)[:1.7.0_17]
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)[:1.7.0_17]
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)[:1.7.0_17]
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)[:1.7.0_17]
at java.lang.StringBuilder.append(StringBuilder.java:132)[:1.7.0_17]
at com.mongodb.ServerDescription.getShortDescription(ServerDescription.java:467)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.ClusterDescription.getShortDescription(ClusterDescription.java:191)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.BaseCluster.updateDescription(BaseCluster.java:158)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster.updateDescription(MultiServerCluster.java:240)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster.onChange(MultiServerCluster.java:149)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster.access$100(MultiServerCluster.java:40)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.MultiServerCluster$DefaultServerStateListener.stateChanged(MultiServerCluster.java:111)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.DefaultServer$DefaultServerStateListener.stateChanged(DefaultServer.java:104)[115:org.db.mongo:1.0.0.201407140926]
at com.mongodb.ServerMonitor$ServerMonitorRunnable.run(ServerMonitor.java:114)[115:org.db.mongo:1.0.0.201407140926]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
2014-07-16 12:19:51,458 | ERROR | qtp1863802945-55 | ContainerResponse | .spi.container.ContainerResponse 406 | 92 - com.sun.jersey.jersey-server - 1.15.0 | The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: Java heap space
at sun.nio.cs.UTF_8.newDecoder(UTF_8.java:68)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:131)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:122)[:1.7.0_17]
at java.lang.StringCoding.decode(StringCoding.java:187)[:1.7.0_17]
at java.lang.String.<init>(String.java:416)[:1.7.0_17]
at org.bson.BasicBSONDecoder$BSONInput.readUTF8String(BasicBSONDecoder.java:544)
at org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:230)
at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:154)
at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:132)
at com.mongodb.DefaultDBDecoder.decode(DefaultDBDecoder.java:62)
at com.mongodb.Response.<init>(Response.java:85)
at com.mongodb.DBPort$1.execute(DBPort.java:141)
at com.mongodb.DBPort$1.execute(DBPort.java:135)
at com.mongodb.DBPort.doOperation(DBPort.java:164)
at com.mongodb.DBPort.call(DBPort.java:135)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:292)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:271)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:237)
at com.mongodb.QueryResultIterator.getMore(QueryResultIterator.java:137)
at com.mongodb.QueryResultIterator.hasNext(QueryResultIterator.java:127)
at com.mongodb.DBCursor._hasNext(DBCursor.java:551)
at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
at com.testing.apptest.search.dao.ObjectSearchDao.getObjects(ObjectSearchDao.java:1648)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectList(ObjectDictionaryManagement.java:572)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectDict(ObjectDictionaryManagement.java:358)
at com.testing.apptest.core.service.alertSettings.ThresholdSettingsManagement.getObjectList(ThresholdSettingsManagement.java:239)
at com.testing.apptest.rest.ThresholdSettingsRest.getDeviceList(ThresholdSettingsRest.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[92:com.sun.jersey.jersey-server:1.15.0]
2014-07-16 12:19:51,457 | ERROR | qtp1863802945-57 | ContainerResponse | .spi.container.ContainerResponse 406 | 92 - com.sun.jersey.jersey-server - 1.15.0 | The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-07-16 12:19:52,669 | WARN | qtp1863802945-57 | ServletHandler | lipse.jetty.util.log.JavaUtilLog 70 | 52 - org.eclipse.jetty.util - 7.6.7.v20120910 | /services/notificationCenter
java.lang.reflect.UndeclaredThrowableException
at org.ops4j.pax.web.service.internal.$Proxy11.service(Unknown Source)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:447)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[59:org.eclipse.jetty.security:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:74)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.Server.handle(Server.java:363)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at org.ops4j.pax.web.service.internal.HttpServiceStarted$2.invoke(HttpServiceStarted.java:210)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
... 27 more
Caused by: javax.servlet.ServletException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[93:com.sun.jersey.servlet:1.15.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)[51:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.1.2]
... 32 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2014-07-16 12:19:52,669 | WARN | qtp1863802945-55 | ServletHandler | lipse.jetty.util.log.JavaUtilLog 70 | 52 - org.eclipse.jetty.util - 7.6.7.v20120910 | /services/alertThreshold/F5/WideIp
java.lang.reflect.UndeclaredThrowableException
at org.ops4j.pax.web.service.internal.$Proxy11.service(Unknown Source)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:447)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)[59:org.eclipse.jetty.security:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:117)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374)[60:org.eclipse.jetty.servlet:7.6.7.v20120910]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:74)[70:org.ops4j.pax.web.pax-web-jetty:1.1.3]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.Server.handle(Server.java:363)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)[54:org.eclipse.jetty.http:7.6.7.v20120910]
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[58:org.eclipse.jetty.server:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)[53:org.eclipse.jetty.io:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[52:org.eclipse.jetty.util:7.6.7.v20120910]
at java.lang.Thread.run(Thread.java:722)[:1.7.0_17]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at org.ops4j.pax.web.service.internal.HttpServiceStarted$2.invoke(HttpServiceStarted.java:210)[69:org.ops4j.pax.web.pax-web-runtime:1.1.3]
... 27 more
Caused by: javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[93:com.sun.jersey.servlet:1.15.0]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[93:com.sun.jersey.servlet:1.15.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:806)[51:org.apache.geronimo.specs.geronimo-servlet_2.5_spec:1.1.2]
... 32 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at sun.nio.cs.UTF_8.newDecoder(UTF_8.java:68)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:131)[:1.7.0_17]
at java.lang.StringCoding$StringDecoder.<init>(StringCoding.java:122)[:1.7.0_17]
at java.lang.StringCoding.decode(StringCoding.java:187)[:1.7.0_17]
at java.lang.String.<init>(String.java:416)[:1.7.0_17]
at org.bson.BasicBSONDecoder$BSONInput.readUTF8String(BasicBSONDecoder.java:544)
at org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:230)
at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:154)
at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:132)
at com.mongodb.DefaultDBDecoder.decode(DefaultDBDecoder.java:62)
at com.mongodb.Response.<init>(Response.java:85)
at com.mongodb.DBPort$1.execute(DBPort.java:141)
at com.mongodb.DBPort$1.execute(DBPort.java:135)
at com.mongodb.DBPort.doOperation(DBPort.java:164)
at com.mongodb.DBPort.call(DBPort.java:135)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:292)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:271)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:237)
at com.mongodb.QueryResultIterator.getMore(QueryResultIterator.java:137)
at com.mongodb.QueryResultIterator.hasNext(QueryResultIterator.java:127)
at com.mongodb.DBCursor._hasNext(DBCursor.java:551)
at com.mongodb.DBCursor.hasNext(DBCursor.java:571)
at com.testing.apptest.search.dao.ObjectSearchDao.getObjects(ObjectSearchDao.java:1648)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectList(ObjectDictionaryManagement.java:572)
at com.testing.apptest.core.service.ObjectDictionaryManagement.getObjectDict(ObjectDictionaryManagement.java:358)
at com.testing.apptest.core.service.alertSettings.ThresholdSettingsManagement.getObjectList(ThresholdSettingsManagement.java:239)
at com.testing.apptest.rest.ThresholdSettingsRest.getDeviceList(ThresholdSettingsRest.java:315)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_17]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_17]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_17]
at java.lang.reflect.Method.invoke(Method.java:601)[:1.7.0_17]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[92:com.sun.jersey.jersey-server:1.15.0]


First of all a process isn't supposed to stop after a OOMError. An OutOfMemoryError is a Throwable and therefore can be handled by the virtual machine. In your case the bundle in question did get removed and therefore the memory consumption is ok again. You'll need to restart your application Bundle and maybe the Pax Web one too.
Regarding your exception I've seen that there is an issue with an ArrayCopy, you seem to have a lot of "big" objects in your application, maybe just increasing the Heap-Size is already enough.


You can set the Xmx (JAVA_MAX_MEM and JAVA_MAX_PERM_MEM) in $KARAF_HOME/bin/setenv

Related

apache ignite out of memory exception

I got out of memory exception and ignite got crashed. After going through the ignite logs, in last metrics I could see heap, off-heap memory usage was about 171 MB,70MB respectively and after 10 secs, ignite logs shows out of memory exception. also, other flags in metrics looks ok
Below is the log snippet
[01:04:29,690][INFO][grid-timeout-worker-#22][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=8a034404, uptime=39 days, 15:50:23.086]
^-- Cluster [hosts=1, CPUs=4, servers=1, clients=1, topVer=22, minorTopVer=0]
^-- Network [addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 172.28.230.222], discoPort=47500, commPort=47100]
^-- CPU [CPUs=4, curLoad=0.07%, avgLoad=0.15%, GC=0%]
^-- Heap [used=171MB, free=95.15%, comm=254MB]
^-- Off-heap memory [used=70MB, free=98.02%, allocated=3377MB]
^-- Page memory [pages=17878]
^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%, allocRam=100MB, allocTotal=0MB]
^-- default region [type=default, persistence=true, lazyAlloc=true,
... initCfg=256MB, maxCfg=3177MB, usedRam=70MB, freeRam=97.78%, allocRam=3177MB, allocTotal=69MB]
^-- metastoreMemPlc region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.95%, allocRam=0MB, allocTotal=0MB]
^-- TxLog region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=100MB, allocTotal=0MB]
^-- volatileDsMemPlc region [type=user, persistence=false, lazyAlloc=true,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
^-- Ignite persistence [used=69MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=7, qSize=0]
^-- Striped thread pool [active=0, idle=8, qSize=0]
[01:04:38,584][INFO][db-checkpoint-thread-#104][Checkpointer] Checkpoint started [checkpointId=41e99f38-7359-4af1-945f-61c92d2a5fb7, startPtr=WALPointer [idx=147, fileOff=11684440, len=381549], checkpointBeforeLockTime=9ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=17ms, checkpointLockHoldTime=19ms, walCpRecordFsyncDuration=2ms, writeCheckpointEntryDuration=2ms, splitAndSortCpPagesDuration=0ms, pages=9, reason='timeout']
[01:04:38,619][SEVERE][db-checkpoint-thread-#104][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteCheckedException: Compound exception for CountDownFuture.]]
class org.apache.ignite.IgniteCheckedException: Compound exception for CountDownFuture.
at org.apache.ignite.internal.util.future.CountDownFuture.addError(CountDownFuture.java:72)
at org.apache.ignite.internal.util.future.CountDownFuture.onDone(CountDownFuture.java:46)
at org.apache.ignite.internal.util.future.CountDownFuture.onDone(CountDownFuture.java:28)
at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:478)
at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriter.run(CheckpointPagesWriter.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Suppressed: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at sun.nio.ch.SimpleAsynchronousFileChannelImpl.implWrite(Unknown Source)
at sun.nio.ch.AsynchronousFileChannelImpl.write(Unknown Source)
at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.write(AsyncFileIO.java:177)
at org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO$5.run(AbstractFileIO.java:117)
at org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.fully(AbstractFileIO.java:53)
at org.apache.ignite.internal.processors.cache.persistence.file.AbstractFileIO.writeFully(AbstractFileIO.java:115)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.write(FilePageStore.java:748)
at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageReadWriteManagerImpl.write(PageReadWriteManagerImpl.java:116)
at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.write(FilePageStoreManager.java:636)
at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointManager.lambda$new$0(CheckpointManager.java:175)
at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriter$1.writePage(CheckpointPagesWriter.java:266)
at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.copyPageForCheckpoint(PageMemoryImpl.java:1343)
at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.checkpointWritePage(PageMemoryImpl.java:1250)
at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriter.writePages(CheckpointPagesWriter.java:207)
at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointPagesWriter.run(CheckpointPagesWriter.java:151)
... 3 more
[01:04:38,620][SEVERE][db-checkpoint-thread-#104][FailureProcessor] No deadlocked threads detected.
[01:04:38,749][SEVERE][db-checkpoint-thread-#104][FailureProcessor] Thread dump at 2022/02/06 01:04:38 CST

unable to create new native thread
This seems to be a non-Ignite exception and most likely is about your system configuration.
Check your Process File Descriptor Limit by running the ulimit -a command and increase it if required. The recommended value is 32768 or above. If it requires an adjustment that can be accomplished by either running ulimit -n 32768 -u 32768 or by modifying the /etc/security/limits.conf

AcquireJobsRunnableImpl trows PSQLException: SSL error: readHandshakeRecord

With a successfully migrated Alfresco (5.2 to) 7.0 repository, I get PSQLException: SSL error: readHandshakeRecord (see stracktrace below) every morning at 4:00, which then causes the repository to stop responding.
Could someone please help me decipher this stack trace? Why is this job running around 4am? I can't find a suitable quartz job. Does anyone know how to manually force this call to fix this problem? At first I thought it might be related to the contentStoreCleaner running at 4:00 am, but disabling this job doesn't change anything.
The only work around I found so far was to disable the activity workflow engine.
2021-08-08 04:35:39,396 ERROR [org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl] [Thread-46] exception during job acquisition: Could not open JDBC Connection for transaction; nested exception is org.postgresql.util.PSQLException: SSL error: readHandshakeRecord
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.postgresql.util.PSQLException: SSL error: readHandshakeRecord
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:309)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl.run(AcquireJobsRunnableImpl.java:54)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.postgresql.util.PSQLException: SSL error: readHandshakeRecord
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:43)
at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:534)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
at org.postgresql.Driver.makeConnection(Driver.java:465)
at org.postgresql.Driver.connect(Driver.java:264)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:265)
... 9 more
Caused by: javax.net.ssl.SSLException: readHandshakeRecord
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1335)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:411)
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41)
... 22 more
Suppressed: java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:380)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450)
... 24 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.flush(SSLSocketOutputRecord.java:251)
at java.base/sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:89)
at java.base/sun.security.ssl.Finished$T13FinishedProducer.onProduceFinished(Finished.java:679)
at java.base/sun.security.ssl.Finished$T13FinishedProducer.produce(Finished.java:658)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
at java.base/sun.security.ssl.Finished$T13FinishedConsumer.onConsumeFinished(Finished.java:1011)
at java.base/sun.security.ssl.Finished$T13FinishedConsumer.consume(Finished.java:874)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1418)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1324)
... 25 more

The stacktrace was misleading - the jdbc connection problem was caused by memory filling up by the Alfresco trashcan cleaner module: OutOfMemoryError: Java heap space due to no limit on getChildAssocs.
We have ~ 4 million nodes in the trashcan and the module retrieves in every batch run all nodes again and again until the memory has been filled up ...

Cannot set Context in open-xchange

Hello everybody when i run:
/opt/open-xchange/sbin/createcontext -A oxadminmaster -P pw_for_master -c 1 -u oxadmin -d "Context Admin" -g Admin -s User -p pw_for_admin -L defaultcontext -e oxadmin#myDomain -q 1024 --access-combination-name=all
following error occurs:
context 1 could not be created:
Server response:
No suitable database available to complete the operation. Please ensure registered databases are reachable and their schemas have enough capacity left and are up-to-date.; exceptionId 321429037-66
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorage.getNextDBHandleByWeight(OXUtilMySQLStorage.java:4080)
at com.openexchange.admin.plugin.hosting.storage.mysqlStorage.OXContextMySQLStorage.create(OXContextMySQLStorage.java:1648)
at com.openexchange.admin.plugin.hosting.rmi.impl.OXContext.createmaincall(OXContext.java:1240)
at com.openexchange.admin.rmi.impl.OXContextCommonImpl.createcommon(OXContextCommonImpl.java:178)
at com.openexchange.admin.plugin.hosting.rmi.impl.OXContext.create(OXContext.java:462)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.create(Unknown Source)
at com.openexchange.admin.console.context.Create.maincall(Create.java:150)
at com.openexchange.admin.console.context.CreateCore.commonfunctions(CreateCore.java:149)
at com.openexchange.admin.console.context.Create.execute(Create.java:94)
at com.openexchange.admin.console.context.Create.main(Create.java:85)
I did run
/opt/open-xchange/sbin/initconfigdb --configdb-pass=db_password -a --mysql-root-passwd=root_password
/opt/open-xchange/sbin/oxinstaller --no-license --servername=oxserver --configdb-user=openexchange --configdb-writehost=localhost --network-listener-host=localhost --servermemory 2048 --master-pass=pw_for_master --configdb-pass=pw_for_db --network-listener-host=localhost --ajp-bind-port=localhost
And could register:
database (groupware database)
server
filestore;
but i am not quite sure, what i registered as database, because with show databases;
i do not find my specified database (name is oxdatabase) in the results.
+--------------------+
| Database |
+--------------------+
| configdb |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.001 sec)
When i created oxdatabase manually and ran
/opt/open-xchange/sbin/createschemas -A oxadminmaster -P pw_for_master -n oxdatabase
but this resulted in:
java.sql.SQLException: Access denied for user 'openexchange'#'localhost' (using password: YES); exceptionId 321429037-68
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorageCommon.getSimpleSQLConnectionFor(OXUtilMySQLStorageCommon.java:137)
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorageCommon.listDatabases(OXUtilMySQLStorageCommon.java:156)
at com.openexchange.admin.storage.mysqlStorage.OXUtilMySQLStorage.createDatabaseSchemas(OXUtilMySQLStorage.java:1389)
at com.openexchange.admin.rmi.impl.OXUtil.createSchemas(OXUtil.java:475)
at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:235)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:180)
at com.sun.proxy.$Proxy0.createSchemas(Unknown Source)
at com.openexchange.admin.console.util.database.CreateSchemas$1.run(CreateSchemas.java:97)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
I checked the settings: PW is ok; user is ok; grants should be ok (i granted star.star);
On disk about 90 GB left; only one partition.
OS: Debian 10
I run as root
So my questions are:
Does my error occur because of my (missing) database?
If yes could someone provide me with the schema for the groupware database?
And why does the error in createschemas occur?

I could fix it.
First step - maybe optional
Re-install Buster
Secondly change hostname to server name -> e.g oxserver

How can I solve this problem loading Grakn schema and data at localhost:48555?

I'm using grakn core 1.8.4 with Windows 10. The grakn server and grakn storage are starting up normally, but when trying to load a schema, Grakn returns the following error message:
Unable to create connection to Grakn instance at localhost:48555
Cause: io.grpc.StatusRuntimeException
UNKNOWN: ID block allocation on partition(30)-namespace(0) timed out in 2.000 min. Please check server logs for the stack trace.
I already checked and there are no other processes listening to the same port. I also disabled the Firewall and that did not solve the problem. Does anyone have any indication of how I should proceed?
Below is part of the log:
2021-01-05 14:19:24,387 [JanusGraphID(30)(0)[0]] WARN g.c.g.d.i.ConsistentKeyIDAuthority - Temporary storage exception while acquiring id block - retrying in PT0.32S: {}
grakn.core.graph.diskstorage.TemporaryBackendException: Wrote claim for id block [1, 10001) in PT0.016S => too slow, threshold is: PT0.01S
at grakn.core.graph.diskstorage.idmanagement.ConsistentKeyIDAuthority.getIDBlock(ConsistentKeyIDAuthority.java:320)
at grakn.core.graph.graphdb.database.idassigner.StandardIDPool$IDBlockGetter.call(StandardIDPool.java:262)
at grakn.core.graph.graphdb.database.idassigner.StandardIDPool$IDBlockGetter.call(StandardIDPool.java:232)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2021-01-05 14:19:24,688 [grpc-request-handler-1] ERROR grakn.core.server.rpc.SessionService - An error has occurred
grakn.core.graph.core.JanusGraphException: ID block allocation on partition(30)-namespace(0) timed out in 2.000 min
at grakn.core.graph.graphdb.database.idassigner.StandardIDPool.waitForIDBlockGetter(StandardIDPool.java:146)
at grakn.core.graph.graphdb.database.idassigner.StandardIDPool.nextBlock(StandardIDPool.java:165)
at grakn.core.graph.graphdb.database.idassigner.StandardIDPool.nextID(StandardIDPool.java:185)
at grakn.core.graph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:334)
at grakn.core.graph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:184)
at grakn.core.graph.graphdb.database.idassigner.VertexIDAssigner.assignID(VertexIDAssigner.java:154)
at grakn.core.graph.graphdb.database.StandardJanusGraph.assignID(StandardJanusGraph.java:416)
at grakn.core.graph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:636)
at grakn.core.graph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:653)
at grakn.core.graph.graphdb.transaction.StandardJanusGraphTx.addVertex(StandardJanusGraphTx.java:649)
at grakn.core.concept.structure.ElementFactory.addVertexElement(ElementFactory.java:104)
at grakn.core.concept.manager.ConceptManagerImpl.addTypeVertex(ConceptManagerImpl.java:188)
at grakn.core.server.session.TransactionImpl.createMetaConcepts(TransactionImpl.java:1297)
at grakn.core.server.session.SessionImpl.initialiseMetaConcepts(SessionImpl.java:123)
at grakn.core.server.session.SessionImpl.<init>(SessionImpl.java:85)
at grakn.core.server.session.SessionFactory.session(SessionFactory.java:115)
at grakn.core.server.rpc.ServerOpenRequest.open(ServerOpenRequest.java:40)
at grakn.core.server.rpc.SessionService.open(SessionService.java:122)
at grakn.protocol.session.SessionServiceGrpc$MethodHandlers.invoke(SessionServiceGrpc.java:339)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:814)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException: null
at java.util.concurrent.FutureTask.get(Unknown Source)
at grakn.core.graph.graphdb.database.idassigner.StandardIDPool.waitForIDBlockGetter(StandardIDPool.java:126)
... 26 common frames omitted

If you get this error, you can try to increase the wait time with the following two parameters in the grakn.properties file:
# The number of milliseconds that the JanusGraph id pool manager will wait before
# giving up on allocating a new block of ids
root.ids.renew-timeout=10
# The number of milliseconds the system waits for an ID block reservation to be acknowledged by the storage backend
ids.authority.wait-time=10

I solved this by increasing the below parameters from 10 o 100 in the .\grakn\server\conf\grakn.properties configuration file.
Not sure if this affects performances.
# The number of milliseconds that the JanusGraph id pool manager will wait before
# giving up on allocating a new block of ids
root.ids.renew-timeout=100
# The number of milliseconds the system waits for an ID block reservation to be acknowledged by the storage backend
ids.authority.wait-time=100

DSE graph not able to create search index asText for a property

I've just started my journey with DSE graph (had fair bit of understanding of Titan earlier). I've set-up DSE graph with Datastax 5.0.3.
When trying to create a search index for a property, I am getting following exception.
schema.vertexLabel('Employee').index('search').search().by('story').asText().add()
org.apache.tinkerpop.gremlin.driver.exception.ResponseException: Cannot create search index with workload: Analytics
I was able to create properties, materialized and secondary indices. But when I tried to create the search index, I am facing this issue.
I realized that while bringing up my single node cluster, I've had to turn off -s flag because that is not letting me to bring up the DSE server. There was some exception when bringing up the node for the first time I was not supposed to set the -s flag as per some Datastax developer QAs.
entrypoint: ["/usr/local/bin/dse-entrypoint", "-k", "-g"]
Now when I tried to enable the -s flag, my node is not coming up and I am getting the following exception.
dse | WARN 12:54:28,038 CLibrary.java:163 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
dse | WARN 12:54:28,038 StartupChecks.java:118 - jemalloc shared library could not be preloaded to speed up memory allocations
dse | WARN 12:54:28,039 StartupChecks.java:150 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
dse | WARN 12:54:28,047 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true
dse | ERROR 12:54:28,710 CassandraDaemon.java:709 - Cannot start node if snitch's data center (SearchGraphAnalytics) differs from previous data center (GraphAnalytics). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
dse | INFO 12:54:28,717 DseDaemon.java:556 - DSE shutting down...
dse | INFO 12:54:28,718 PluginManager.java:104 - All plugins are stopped.
dse | Oct 27, 2016 12:54:28 PM org.apache.coyote.http11.Http11Protocol pause
dse | INFO: Pausing Coyote HTTP/1.1 on http-172.19.0.3-8983
dse | Oct 27, 2016 12:54:29 PM org.apache.catalina.core.StandardService stop
dse | INFO: Stopping service Solr
dse | INFO 12:54:29,907 SolrHttpAuditLogFilter.java:225 - Shutting down Solr audit logging filter
dse | INFO 12:54:29,924 RepeatablePOSTQueryFilter.java:81 - Shutting down com.datastax.bdp.search.solr.RepeatablePOSTQueryFilter filter
dse | Oct 27, 2016 12:54:29 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
dse | SEVERE: The web application [/solr] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak.
dse | Oct 27, 2016 12:54:29 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
dse | SEVERE: The web application [/solr] appears to have started a thread named [NonPeriodicTasks:1] but has failed to stop it. This is very likely to create a memory leak.
dse | Oct 27, 2016 12:54:29 PM org.apache.coyote.http11.Http11Protocol destroy
dse | INFO: Stopping Coyote HTTP/1.1 on http-172.19.0.3-8983
dse | INFO 12:54:33,191 MessageServer.java:129 - internode-messaging message server finished shutting down.
dse | INFO 12:54:37,209 MessageServer.java:129 - internode-messaging message server finished shutting down.
dse | Exception in thread "Daemon shutdown" java.lang.AssertionError
dse | at org.apache.cassandra.gms.Gossiper.addLocalApplicationStateInternal(Gossiper.java:1427)
dse | at org.apache.cassandra.gms.Gossiper.addLocalApplicationStates(Gossiper.java:1451)
dse | at org.apache.cassandra.gms.Gossiper.addLocalApplicationState(Gossiper.java:1441)
dse | at com.datastax.bdp.gms.DseState.setActiveStatusSync(DseState.java:241)
dse | at com.datastax.bdp.server.DseDaemon.preStop(DseDaemon.java:576)
dse | at com.datastax.bdp.server.DseDaemon.safeStop(DseDaemon.java:587)
dse | at com.datastax.bdp.server.DseDaemon.lambda$getShutdownHook$226(DseDaemon.java:905)
dse | at java.lang.Thread.run(Thread.java:745)
Please suggest how I can rectify this situation and be able to add the searchIndex to my properties.

This error is stating that you are starting up the cluster with a different name than what the cluster expects based on a previous configuration/start.
By default, unless you override the cluster name in the cassandra.yaml file, your cluster will startup with a name based on the workloads you enable, i.e. -s, -t. In your case, since you started the cluster as an Analytics cluster and then restarted it as a SearchAnalytics cluster, the cluster name is defaulting to the new cluster name, which doesn't match the old cluster name.
The easiest thing to do here is to wipe your cassandra commit log, caches, and data directory and restart the node. That will wipe out the old cluster name from your system tables and will allow the cluster to start. Doing this will wipe out any data you had in the cluster.

Resources