I configured a new pipeline in HERE OLP. Having issues while running it. If anybody has past experience dealing with here olp pipeline, please advise.
2019-08-16 18:21:00.946 [SIGTERM handler] level=ERROR org.apache.spark.executor.CoarseGrainedExecutorBackend - RECEIVED SIGNAL TERM // PipeLineId=63e07977-ba0a-4997-a633-1b578ae86ce1 Realm=olp-ford DeploymentId=4832de82-afb9-4f03-b92a-a6ba12eb9116 VersionId=0bc6df20-8873-4195-bfdb-87271a95fdba
3-4195-bfdb-87271a95fdba
com.here.platform.data.processing.exception.ContextRuntimeException: // Task=seedmap cat=highways l=adas-attributes p=18939748
at com.here.platform.data.processing.exception.ContextRuntimeException$.apply(ContextRuntimeException.scala:46)
at com.here.platform.data.processing.exception.package$.wrapInTryWithContext(package.scala:109)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1$$anonfun$apply$5.apply(DirectCompileFnUtils.scala:83)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1$$anonfun$apply$5.apply(DirectCompileFnUtils.scala:82)
at com.here.platform.data.processing.logging.Implicits$LogContextKeyWrapper$.withChild$extension(Implicits.scala:46)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1.apply(DirectCompileFnUtils.scala:82)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1.apply(DirectCompileFnUtils.scala:81)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: requirement failed: Compile function returned partition key not compatible with the layer: Key('output,'seed-map,18939748)
at scala.Predef$.require(Predef.scala:224)
at com.here.platform.data.processing.driver.impl.executors.ExecutorUtils$.verifyPartitionKey(ExecutorUtils.scala:291)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1$$anonfun$apply$5$$anonfun$apply$6$$anonfun$apply$7.apply(DirectCompileFnUtils.scala:85)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1$$anonfun$apply$5$$anonfun$apply$6$$anonfun$apply$7.apply(DirectCompileFnUtils.scala:85)
at scala.collection.immutable.List.foreach(List.scala:381)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1$$anonfun$apply$5$$anonfun$apply$6.apply(DirectCompileFnUtils.scala:85)
at com.here.platform.data.processing.driver.impl.executors.DirectCompileFnUtils$$anonfun$applyMappingFn$1$$anonfun$apply$5$$anonfun$apply$6.apply(DirectCompileFnUtils.scala:83)
at scala.util.Try$.apply(Try.scala:192)
at com.here.platform.data.processing.exception.package$.wrapInTryWithContext(package.scala:100)
... 17 common frames omitted
The problem might be related to the partition type of layer. Make sure the returned key has the correct Partition.Name type. If your layer is generically partitioned you have to build the partition name with Generic(), and not HereTile().
Also check Partition Type and Zoom level of layer.
Related
Error occurred during initialization of boot layer
java.lang.module.FindException: Unable to derive module descriptor for C:\JavaFX\jlfgr-1_0.jar
Caused by: java.lang.IllegalArgumentException: jlfgr.1.0: Invalid module name: '1' is not a Java identifier
I am getting this error again and again. I had tried all the solutions which are available on this site but my bug was not fixed. May any one please help me?
I've read the spring-kafka documentation, examples I found, and half of stackoverflow, but I fail to understand, who #EmbeddedKafka should work. Especially for integration tests.
#RunWith(SpringRunner.class)
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
#EnableKafka
#EmbeddedKafka(controlledShutdown = true
// brokerProperties = {"log.dir=${kafka.broker.logs-dir}",
// "listeners=PLAINTEXT://localhost:${kafka.broker.port:3333}",
// "auto.create.topics.enable=${kafka.broker.topics-enable:true}"}
)
and I had to add into test application.properties following:
….kafka.bootstrap-servers=${spring.embedded.kafka.brokers}
with all this in place, when test it executed, it complains several lines about kafka not being accessible:
[Producer clientId=producer-2] Connection to node 0 could not be established. Broker may not be available.
but after half of screen it proceeds, test succeds and finished with last unplesant message:
2019-06-17 12:31:54.438 WARN [testing,,,] 32448 --- [ost-startStop-1] o.a.c.loader.WebappClassLoaderBase : The web application [testing] appears to have started a thread named [kafka-producer-network-thread | producer-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.kafka.common.network.Selector.select(Selector.java:674)
org.apache.kafka.common.network.Selector.poll(Selector.java:396)
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
java.lang.Thread.run(Thread.java:748)
questions:
1/ why there are those messages about broker inavailability at the start of each method, even if the context wasn't dirties and reintialized?
2/ why is the ugly message with unclosed thread present? What's wrong?
After running a groovy script as task to create a role with:
security.addRole(// id
roleDeveloper,
// name
roleDeveloper,
// description
"A developer on ${repoCap} group",
// privileges
["nx-repository-view-maven2-${repo}-dependencies-browse",
"nx-repository-view-maven2-${repo}-dependencies-read"],
// roles
["dw-all-public-repos"])
I can't access to the roles menu. I get the following error:
com.orientechnologies.orient.core.exception.ODatabaseException: Error on deserialization of Serializable DB name="security"
[...]
Caused by: java.lang.ClassNotFoundException: org.codehaus.groovy.runtime.GStringImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) [na:1.8.0_91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) [na:1.8.0_91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [na:1.8.0_91]
at org.apache.felix.framework.BundleWiringImpl.doImplicitBootDelegation(BundleWiringImpl.java:1782) [na:na]
at org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:1717) [na:na]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1552) [na:na]
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:79) [na:na]
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:2018) [na:na]
After running several tests (with and without String interpolations) on several version of Nexus (3.x) it looks like String interpolations are supported for some parameters but not for privileges parameter.
Is it a known issue ?
Now that my Roles menu is inaccessible due to this above error is there a way to fix it ? (I tried to remove it with a script but it failed because delete perform a load first)
Sorry for the problems Alexandre. It looks like you'll have to connect to the database directly in order to fix the problematic records. Instructions for how to do this with Nexus offline are here: https://support.sonatype.com/hc/en-us/articles/115002930827-Accessing-the-OrientDB-Console
In particular the database you're looking to connect to is 'security':
connect plocal:data/db/security admin admin
And the tables you will need to inspect/delete from are 'privilege' and 'role'.
I'll keep an eye out here in case you run into problems or have any followup questions.
I am using solr 4.10.2. I tried to perform a shard split on my solr cloud test cluster. It fails all the time if the index type is set to "native" or "simple".
Is that normal? I can perform shard splitting if the index type is set to "single" or "none".
They advertise that shard splitting can be done while solr is running and i hardly imagine poking around changing the lock type of a production server...
Here is the test environment:
1 shard, 2 nodes, 1 collection.
Initially the collection was empty. I added few documents, verified that they have been replicated. All worked.
Issued the split shard command:
server1:port/solr/admin/collections?action=SPLITSHARD&collection=mycollection&shard=shard1&async=myhandle
After verifying that the operation had finished, by calling
server1:port/solr/admin/collections?action=REQUESTSTATUS&requestid=myhandle
The status was "complete".
Here is the log:
OverseerCollectionProcessor.processMessage : splitshard , {
"operation":"splitshard",
"shard":"shard1",
"collection":"mycollection",
"async":"myhandle"}
1/26/2015, 1:49:02 PM
ERROR
CoreContainer
Error creating core [mycollection_shard1_0_replica1]: Error opening new searcher
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:873)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:466)
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:575)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:199)
at org.apache.solr.handler.admin.CoreAdminHandler$ParallelCoreAdminHandlerThread.run(CoreAdminHandler.java:1234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
... 9 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock#/nfs/solr/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:89)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1528)
... 11 more
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:873)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:466)
at org.apache.solr.handler.admin.CoreAdminHandler.handleCreateAction(CoreAdminHandler.java:575)
at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestInternal(CoreAdminHandler.java:199)
at org.apache.solr.handler.admin.CoreAdminHandler$ParallelCoreAdminHandlerThread.run(CoreAdminHandler.java:1234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1565)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:845)
... 9 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock#/nfs/solr/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:89)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:753)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:279)
at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:111)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1528)
... 11 more
1/26/2015, 1:49:26 PM
ERROR
SolrIndexWriter
SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
1/26/2015, 1:49:26 PM
ERROR
SolrIndexWriter
Error closing IndexWriter
java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3230)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3203)
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:907)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:984)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:954)
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:129)
at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:182)
at java.lang.System$2.invokeFinalize(System.java:1213)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
I fixed the problem. Here is how:
When i created the solr cloud environment, i used the -Dsolr.data.dir property to map the collection storage to a different file system. This was because i was running VMs with limited storage capacity. Once i removed this property everything started working.
I think solr tries to use the same solr.data.dir path for the new cores created by the shard split causing the lock problem.
I have configured Cassandra-1.2.2 in cluster mode. But while starting Titan with my cassandra configuration, it Showing the following exception
Exception in thread "main" java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager
at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:268)
at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:226)
at com.thinkaurelius.titan.diskstorage.Backend.<init>(Backend.java:97)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:406)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:62)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:40)
at com.thinkaurelius.titan.tinkerpop.rexster.RexsterTitanServer.start(RexsterTitanServer.java:70)
at com.thinkaurelius.titan.tinkerpop.rexster.RexsterTitanServer.startDaemon(RexsterTitanServer.java:80)
at com.thinkaurelius.titan.tinkerpop.rexster.RexsterTitanServer.main(RexsterTitanServer.java:118)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:257)
... 8 more
Caused by: com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure in storage backend
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:394)
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:164)
... 13 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0] No hosts to borrow from
at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.<init>(RoundRobinExecuteWithFailover.java:31)
at com.netflix.astyanax.connectionpool.impl.TokenAwareConnectionPoolImpl.newExecuteWithFailover(TokenAwareConnectionPoolImpl.java:74)
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229)
at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:131)
at com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:252)
at com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:389)
... 14 more
Please help to find what is wrong with?
Sometimes there are Astyanax connections issues on AWS. If you are running this on AWS, try changing the configuration to:
storage.backend = cassandrathrift