I have something which looks like that in my "initialize" method :
SpinnerValueFactory<Double> betaValueFactory = new SpinnerValueFactory.DoubleSpinnerValueFactory(-9999999, 9999999, 0);
this.betaSpinner.setValueFactory(betaValueFactory);
SpinnerValueFactory<Integer> temperatureValueFactory = new SpinnerValueFactory.IntegerSpinnerValueFactory(-9999999, 9999999, 0);
this.temperatureSpinner.setValueFactory(temperatureValueFactory);
The problem is that when the user types a wrong thing like "aaa" in one of the spinners, I have an exception like this one :
Exception in thread "JavaFX Application Thread" java.lang.NumberFormatException: For input string: "aaa"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:68)
at java.base/java.lang.Integer.parseInt(Integer.java:658)
at java.base/java.lang.Integer.valueOf(Integer.java:989)
at javafx.base/javafx.util.converter.IntegerStringConverter.fromString(IntegerStringConverter.java:49)
at javafx.base/javafx.util.converter.IntegerStringConverter.fromString(IntegerStringConverter.java:35)
at javafx.controls/javafx.scene.control.Spinner.commitValue(Spinner.java:460)
at javafx.controls/javafx.scene.control.Spinner.lambda$new$3(Spinner.java:168)
at javafx.base/com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent(ExpressionHelper.java:348)
at javafx.base/com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:80)
at javafx.base/javafx.beans.property.ReadOnlyBooleanPropertyBase.fireValueChangedEvent(ReadOnlyBooleanPropertyBase.java:72)
at javafx.graphics/javafx.scene.Node$FocusedProperty.notifyListeners(Node.java:8159)
at javafx.graphics/javafx.scene.Node.setFocused(Node.java:8212)
at javafx.graphics/javafx.scene.Scene$KeyHandler.setWindowFocused(Scene.java:4038)
at javafx.graphics/javafx.scene.Scene$KeyHandler.lambda$new$0(Scene.java:4060)
at javafx.base/com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent(ExpressionHelper.java:348)
at javafx.base/com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:80)
at javafx.base/javafx.beans.property.ReadOnlyBooleanPropertyBase.fireValueChangedEvent(ReadOnlyBooleanPropertyBase.java:72)
at javafx.base/javafx.beans.property.ReadOnlyBooleanWrapper.fireValueChangedEvent(ReadOnlyBooleanWrapper.java:103)
at javafx.base/javafx.beans.property.BooleanPropertyBase.markInvalid(BooleanPropertyBase.java:111)
at javafx.base/javafx.beans.property.BooleanPropertyBase.set(BooleanPropertyBase.java:145)
at javafx.graphics/javafx.stage.Window.setFocused(Window.java:674)
at javafx.graphics/javafx.stage.Window$1.setFocused(Window.java:149)
at javafx.graphics/com.sun.javafx.stage.WindowHelper.setFocused(WindowHelper.java:112)
at javafx.graphics/com.sun.javafx.stage.WindowPeerListener.changedFocused(WindowPeerListener.java:64)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassWindowEventHandler.run(GlassWindowEventHandler.java:126)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassWindowEventHandler.run(GlassWindowEventHandler.java:40)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassWindowEventHandler.lambda$handleWindowEvent$4(GlassWindowEventHandler.java:176)
at javafx.graphics/com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:412)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassWindowEventHandler.handleWindowEvent(GlassWindowEventHandler.java:174)
at javafx.graphics/com.sun.glass.ui.Window.handleWindowEvent(Window.java:1351)
at javafx.graphics/com.sun.glass.ui.Window.notifyFocus(Window.java:1330)
at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
at java.base/java.lang.Thread.run(Thread.java:830)
I would like solve this problem by forcing the user to type a correct value by displaying an alert with a little message but I don't know how to do it, that's why I need you.
Thanks for your help.
Related
When I try to do a POST request using khttp, I get this error:
E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.emir.getirme, PID: 9379
android.os.NetworkOnMainThreadException
at android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode.java:1448)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:355)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:356)
at java.net.Socket.connect(Socket.java:616)
at com.android.okhttp.internal.Platform.connectSocket(Platform.java:145)
at com.android.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:141)
at com.android.okhttp.internal.io.RealConnection.connect(RealConnection.java:112)
at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:184)
at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:126)
at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:95)
at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:281)
at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:224)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:461)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:407)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaders(HttpURLConnectionImpl.java:163)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getHeaderFields(HttpURLConnectionImpl.java:223)
at khttp.responses.GenericResponse$Companion.getCookieJar$khttp(GenericResponse.kt:35)
at khttp.responses.GenericResponse$Companion$defaultEndInitializers$3.invoke(GenericResponse.kt:116)
at khttp.responses.GenericResponse$Companion$defaultEndInitializers$3.invoke(GenericResponse.kt:32)
at khttp.responses.GenericResponse$connection$2.invoke(GenericResponse.kt:164)
at khttp.responses.GenericResponse$connection$2.invoke(GenericResponse.kt:30)
at khttp.responses.GenericResponse.openRedirectingConnection$khttp(GenericResponse.kt:124)
at khttp.responses.GenericResponse.getConnection(GenericResponse.kt:163)
at khttp.responses.GenericResponse.getRaw(GenericResponse.kt:207)
at khttp.responses.GenericResponse.getContent(GenericResponse.kt:216)
at khttp.responses.GenericResponse.init$khttp(GenericResponse.kt:350)
at khttp.KHttp.request(KHttp.kt:59)
at khttp.KHttp.post(KHttp.kt:48)
at khttp.KHttp.post$default(KHttp.kt:47)
at com.emir.getirme.MainActivity.onCreate$lambda-1(MainActivity.kt:51)
at com.emir.getirme.MainActivity.lambda$3IBZyc9kPzUSvfk0gsgNPVZ2ETU(Unknown Source:0)
at com.emir.getirme.-$$Lambda$MainActivity$3IBZyc9kPzUSvfk0gsgNPVZ2ETU.onClick(Unknown Source:2)
at android.view.View.performClick(View.java:6256)
at com.google.android.material.button.MaterialButton.performClick(MaterialButton.java:1119)
at android.view.View$PerformClick.run(View.java:24701)
at android.os.Handler.handleCallback(Handler.java:789)
at android.os.Handler.dispatchMessage(Handler.java:98)
at android.os.Looper.loop(Looper.java:164)
at android.app.ActivityThread.main(ActivityThread.java:6541)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:240)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:767)
This is my code:
khttp.post(
"http://127.0.0.1:5000/order",
headers = mapOf(
"address" to address.toString(),
"items" to Json.encodeToString(listOf("Su x1", "Gofret x5", "Dondurma x1")),
"price" to "100",
"orderer" to name.toString(),
"phone" to phone.toString()
),
)
I'm trying to send a request to a Flask server running in my local PC. The server is not the problem as I tested it with Python.
The code is not inside an async task or any type of async function. I tried to use khttp.async but I didn't have khttp.async for some reason.
Even though the exception message is very clear, If you want to understand in detail why this happens, please read
As to how you can fix this, you can use kotln coroutines for this, once you have included the given dependencies in your build, you can do the following
// Start network request on a background thread using coroutine
lifecycleScope.launch(Dispatchers.IO) {
khttp.post(
"http://127.0.0.1:5000/order",
headers = mapOf(
"address" to address.toString(),
"items" to Json.encodeToString(listOf("Su x1", "Gofret x5", "Dondurma x1")),
"price" to "100",
"orderer" to name.toString(),
"phone" to phone.toString()
),
)
}
I have a constraint that crashes in the latest OptaPlanner 8.0.0, but used to work fine on 7.46.0.
As expected, IntelliJ's code inspection (and the debugger) shows that after the first join, the stream is a TriConstraintStream. The runtime class makes more sense to me than the class OptaPlanner is trying to cast to.
When leaving out the last groupBy the error goes away, so that clause seems to cause the issue.
Did something change in the way join and groupby worked?
It seems that the underlying OptaPlanner code was refactored for 8.0.0, so I have trouble seeing what exactly changed in OptaPlanner.
Should I add something to ensure that a TriJoin is used instead of a BiJoin?
I could not find any relevant notes in the migration documentation.
protected Constraint preventProductionShortage(ConstraintFactory factory) {
return factory.from(Demand.class)
.groupBy(Demand::getSKU,
Demand::getWeekNumber
)//BiConstraintStream
.join(Demand.class,
equal((sku, weekNumber)-> sku, Demand::getSKU),
greaterThanOrEqual((sku, weekNumber)-> weekNumber, Demand::getWeekNumber)//TriConstraintStream
)
.groupBy((sku, weekNumber, totalDemand) -> sku,
(sku, weekNumber, totalDemand) -> weekNumber,
sum((sku, weekNumber, totalDemand) -> totalDemand.getOrderQuantity())
)//TriConstraintStream
.penalize("Penalty", HardMediumSoftScore.ONE_MEDIUM,
(sku_weekNumber, demandQty, productionQty) -> 1);
}
Stack trace:
java.lang.ClassCastException: class org.optaplanner.core.impl.score.stream.tri.CompositeTriJoiner cannot be cast to class org.optaplanner.core.impl.score.stream.bi.AbstractBiJoiner (org.optaplanner.core.impl.score.stream.tri.CompositeTriJoiner and org.optaplanner.core.impl.score.stream.bi.AbstractBiJoiner are in unnamed module of loader 'app')
at org.optaplanner.core.impl.score.stream.drools.common.rules.BiJoinMutator.<init>(BiJoinMutator.java:40)
at org.optaplanner.core.impl.score.stream.drools.common.rules.UniRuleAssembler.join(UniRuleAssembler.java:70)
at org.optaplanner.core.impl.score.stream.drools.common.rules.AbstractRuleAssembler.join(AbstractRuleAssembler.java:179)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintSubTree.getRuleAssembler(ConstraintSubTree.java:94)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintSubTree.getRuleAssembler(ConstraintSubTree.java:89)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintGraph.generateRule(ConstraintGraph.java:431)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintGraph.lambda$generateRule$57(ConstraintGraph.java:423)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.optaplanner.core.impl.score.stream.drools.common.ConstraintGraph.generateRule(ConstraintGraph.java:424)
at org.optaplanner.core.impl.score.stream.drools.DroolsConstraintFactory.buildSessionFactory(DroolsConstraintFactory.java:101)
at org.optaplanner.core.impl.score.director.stream.ConstraintStreamScoreDirectorFactory.<init>(ConstraintStreamScoreDirectorFactory.java:77)
at org.optaplanner.test.impl.score.stream.DefaultConstraintVerifier.verifyThat(DefaultConstraintVerifier.java:63)
at org.optaplanner.test.impl.score.stream.DefaultConstraintVerifier.verifyThat(DefaultConstraintVerifier.java:32)
at com.ohly.planner.constraints.ConstraintsTest.weekShortageSingleSKU(ConstraintsTest.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Process finished with exit code -1
[edit] For completeness, the new function as suggested by Lukáš Petrovický
protected Constraint preventProductionShortage(ConstraintFactory factory) {
return factory.from(Demand.class)
.join(Demand.class,
equal(Demand::getSKU),
greaterThanOrEqual(demand -> demand.getWeekNumber()))
.groupBy((d, d2) -> d.getSKU(),
(d, d2) -> d.getWeekNumber(),
sum((d,d2) -> d2.getOrderQuantity())
)
...
[/edit]
This was an unfortunate bug not caught by the existing test coverage.
The fix is aimed at OptaPlanner 8.0.1, incl. test coverage improvement.
That said, I would argue that the constraint is not very efficient. Unless I'm missing some key implications, the following is semantically very similar, yet much faster:
protected Constraint preventProductionShortage(ConstraintFactory factory) {
return factory.from(Demand.class)
.join(Demand.class,
equal(Demand::getSKU),
greaterThanOrEqual(demand -> demand.getWeekNumber()))
.groupBy(..., ..., sum((demand, demand2) -> ...))
.penalize("Penalty", HardMediumSoftScore.ONE_MEDIUM);
}
Note how I eliminated the first use of groupBy(). There may be some difference though in how many tuples are penalized this way, which may or may not be what you want. Feel free to open another question on that.
Im looking to get started with Usergrid, and am following the steps on https://github.com/apache/incubator-usergrid/tree/master/stack
I get to the point of launching the gui launcher.
However I cannot get the webpage to load by going to http://apigee.github.io/usergrid-portal/?api_url=http://localhost:8080, I get a 404 each time.
What am I doing wrong??
And in the console I get the following error:
2014-12-13 13:12:48,133 ERROR (AWT-EventQueue-0) [org.apache.usergrid.persistence.cassandra.EntityManagerImpl] - Unable to load entity: 00000000-0000-0000-0000-000000000001
java.lang.NullPointerException
at me.prettyprint.hector.api.factory.HFactory.createVirtualKeyspace(HFactory.java:320)
at org.apache.usergrid.persistence.cassandra.CassandraService.getKeyspace(CassandraService.java:261)
at org.apache.usergrid.persistence.cassandra.CassandraService.getApplicationKeyspace(CassandraService.java:274)
at org.apache.usergrid.persistence.cassandra.EntityManagerImpl.getEntityType(EntityManagerImpl.java:996)
at org.apache.usergrid.persistence.cassandra.EntityManagerImpl.getRef(EntityManagerImpl.java:1797)
at org.apache.usergrid.persistence.cassandra.EntityManagerImpl.validate(EntityManagerImpl.java:1757)
at org.apache.usergrid.persistence.cassandra.EntityManagerImpl.validate(EntityManagerImpl.java:1742)
at org.apache.usergrid.persistence.cassandra.RelationManagerImpl.searchCollection(RelationManagerImpl.java:1764)
at org.apache.usergrid.persistence.cassandra.EntityManagerImpl.getUserByIdentifier(EntityManagerImpl.java:1536)
at org.apache.usergrid.persistence.cassandra.EntityManagerImpl$$FastClassByCGLIB$$db0622e.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
at org.apache.usergrid.persistence.cassandra.EntityManagerImpl$$EnhancerByCGLIB$$fe4a7dad.getUserByIdentifier(<generated>)
at org.apache.usergrid.management.cassandra.ManagementServiceImpl.getUserEntityByIdentifier(ManagementServiceImpl.java:1035)
at org.apache.usergrid.management.cassandra.ManagementServiceImpl.getAdminUserByEmail(ManagementServiceImpl.java:1028)
at org.apache.usergrid.launcher.Server.getAccessTokenForAdminUser(Server.java:599)
at org.apache.usergrid.launcher.App.getAccessToken(App.java:268)
at org.apache.usergrid.launcher.LauncherFrame.getAdminURI(LauncherFrame.java:384)
at org.apache.usergrid.launcher.LauncherFrame$5.actionPerformed(LauncherFrame.java:211)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2341)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.Component.processMouseEvent(Component.java:6516)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3320)
at java.awt.Component.processEvent(Component.java:6281)
at java.awt.Container.processEvent(Container.java:2229)
at java.awt.Component.dispatchEventImpl(Component.java:4872)
at java.awt.Container.dispatchEventImpl(Container.java:2287)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
at java.awt.Container.dispatchEventImpl(Container.java:2273)
at java.awt.Window.dispatchEventImpl(Window.java:2719)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:735)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
at java.awt.EventQueue$4.run(EventQueue.java:708)
at java.awt.EventQueue$4.run(EventQueue.java:706)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:705)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
The getting started steps are wrong. You can use the portal by building it (with build.sh in the portal directory) and going to a URL like this in your browser:
file:///{path to your local codebase}/usergrid/portal/index.html#!/login?api_url=http://localhost:8080
Using Spark 1.1
I have 2 datasets. One is very large and the other was reduced (using some 1:100 filtering) to much smaller scale. I need to reduce the large dataset to the same scale, by joining only those items from the smaller list with their corresponding counterparts in the larger list (those lists contain elements that have a mutual join field).
I am doing that using the following code:
The "if(joinKeys != null)" part is the relevant part
Smaller list is "joinKeys", larger list is "keyedEvents"
private static JavaRDD<ObjectNode> createOutputType(JavaRDD jsonsList, final String type, String outputPath,JavaPairRDD<String,String> joinKeys) {
outputPath = outputPath + "/" + type;
JavaRDD events = jsonsList.filter(new TypeFilter(type));
// This is in case we need to narrow the list to match some other list of ids... Recommendation List, for example... :)
if(joinKeys != null) {
JavaPairRDD<String,ObjectNode> keyedEvents = events.mapToPair(new KeyAdder("requestId"));
JavaRDD < ObjectNode > joinedEvents = joinKeys.join(keyedEvents).values().map(new PairToSecond());
events = joinedEvents;
}
JavaPairRDD<String,Iterable<ObjectNode>> groupedEvents = events.mapToPair(new KeyAdder("sliceKey")).groupByKey();
// Add convert jsons to strings and add "\n" at the end of each
JavaPairRDD<String, String> groupedStrings = groupedEvents.mapToPair(new JsonsToStrings());
groupedStrings.saveAsHadoopFile(outputPath, String.class, String.class, KeyBasedMultipleTextOutputFormat.class);
return events;
}
Thing is when running this job, I always get the same error:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:40)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2757 in stage 13.0 failed 4 times, most recent failure: Lost task 2757.3 in stage 13.0 (TID 47681, hadoop-w-175.c.taboola-qa-01.internal): java.io.FileNotFoundException: /hadoop/spark/tmp/spark-local-20141201184944-ba09/36/shuffle_6_2757_2762 (Too many open files)
java.io.FileOutputStream.open(Native Method)
java.io.FileOutputStream.<init>(FileOutputStream.java:221)
org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:123)
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:192)
org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:67)
org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:65)
scala.collection.Iterator$class.foreach(Iterator.scala:727)
scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.scala:65)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
I already increased my ulimits, by doing the following on all cluster machines:
echo "* soft nofile 900000" >> /etc/security/limits.conf
echo "root soft nofile 900000" >> /etc/security/limits.conf
echo "* hard nofile 990000" >> /etc/security/limits.conf
echo "root hard nofile 990000" >> /etc/security/limits.conf
echo "session required pam_limits.so" >> /etc/pam.d/common-session
echo "session required pam_limits.so" >> /etc/pam.d/common-session-noninteractive
But doesn't fix my problem...
The bdutil framework works in a way the user "hadoop" is the one running the job. The script that deploys the cluster, created a file /etc/security/limits.d/hadoop.conf that overrided the ulimit settings for "hadoop" user, which I wasn't aware of. By deleting this file, or alternatively setting the desired ulimits there, I was able to resolve the problem.
As far as I know, pmap in Clojure works just like map, but it calculates results in parallel, using futures under the hood. So it should "just work" with a function and a sequence, if map works with them. (Unless there are evil side effects that prevent it, but in case of my program there is nothing more than loading data from http server and transforming it)
And in my case pmap doesn't work as expected. Why can this happen?
The problem arises here (if I change map to pmap):
https://github.com/magicgoose/DvachMaster/blob/master/src/dvach/core.clj#L82
(defn thread-list
"load threads from all pages, trying each page at most `max-trials` times with `retry-inteval`"
[board]
(try
(let [p0 (load-body (board-addr board 0))
numpages (count (:pages p0))
other-pages (map ; problem here
(comp
load-body
(partial board-addr board))
(range 1 numpages))
all-pages (cons p0 other-pages)
]
(doall
((comp (partial reduce concat) (partial map :threads)) all-pages)))
(catch Throwable e
(.printStackTrace e))))
The exception I get:
java.util.concurrent.ExecutionException: java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.concurrent.Future
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at clojure.core$deref_future.invoke(core.clj:2108)
at clojure.core$future_call$reify__6267.deref(core.clj:6308)
at clojure.core$deref.invoke(core.clj:2128)
at clojure.core$pmap$step__6280$fn__6282.invoke(core.clj:6358)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
at clojure.lang.RT.seq(RT.java:484)
at clojure.core$seq.invoke(core.clj:133)
at clojure.core$map$fn__4207.invoke(core.clj:2479)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.RT.next(RT.java:598)
at clojure.core$next.invoke(core.clj:64)
at clojure.core.protocols$fn__6034.invoke(protocols.clj:146)
at clojure.core.protocols$fn__6005$G__6000__6014.invoke(protocols.clj:19)
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:27)
at clojure.core.protocols$fn__6026.invoke(protocols.clj:53)
at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13)
at clojure.core$reduce.invoke(core.clj:6175)
at clojure.lang.AFn.applyToHelper(AFn.java:163)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:619)
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$comp$fn__4154.invoke(core.clj:2331)
at dvach.core$thread_list.invoke(core.clj:91)
at dvach.core$eval3813.invoke(NO_SOURCE_FILE:2)
at clojure.lang.Compiler.eval(Compiler.java:6619)
at clojure.lang.Compiler.eval(Compiler.java:6582)
at clojure.core$eval.invoke(core.clj:2852)
at clojure.main$repl$read_eval_print__6588$fn__6591.invoke(main.clj:259)
at clojure.main$repl$read_eval_print__6588.invoke(main.clj:259)
at clojure.main$repl$fn__6597.invoke(main.clj:277)
at clojure.main$repl.doInvoke(main.clj:277)
at clojure.lang.RestFn.invoke(RestFn.java:1096)
at clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__1023.invoke(interruptible_eval.clj:56)
at clojure.lang.AFn.applyToHelper(AFn.java:159)
at clojure.lang.AFn.applyTo(AFn.java:151)
at clojure.core$apply.invoke(core.clj:617)
at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1788)
at clojure.lang.RestFn.invoke(RestFn.java:425)
at clojure.tools.nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:41)
at clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__1064$fn__1067.invoke(interruptible_eval.clj:171)
at clojure.core$comp$fn__4154.invoke(core.clj:2330)
at clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__1057.invoke(interruptible_eval.clj:138)
at clojure.lang.AFn.run(AFn.java:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.util.concurrent.Future
at clojure.core$deref_future.invoke(core.clj:2108)
at clojure.core$deref.invoke(core.clj:2129)
at dvach.core$load_body.invoke(core.clj:74)
at clojure.core$comp$fn__4154.invoke(core.clj:2331)
at clojure.core$pmap$fn__6275$fn__6276.invoke(core.clj:6354)
at clojure.core$binding_conveyor_fn$fn__4107.invoke(core.clj:1836)
at clojure.lang.AFn.call(AFn.java:18)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
... 3 more
The problem the stack trace complains about is with #max-trials on line 74; this should read max-trials instead. (max-trials is a loop variable initialized to #retry-count on line 66; it'll be a number then, to be decremented on each iteration.)
It may well arise intermittently, since that point in the code is only reached if the try block starting on line 68 fails to fetch the result.