I decided to build a web service(app) for Apache Spark with Apache Livy.
Livy server is up and running on localhost port 8998 according to Livy configuration defaults.
My test program is a sample application in Apache Livy documentation: https://livy.incubator.apache.org/docs/latest/programmatic-api.html
While creating LivyClient by LivyClientBuilder class,
client = new LivyClientBuilder().setURI(new
URI("http","user:info","localhost",8998,"","",""))
.build();
I got "URI is not supported by any registered client factories" exception:
Exception in thread "main" java.lang.IllegalArgumentException: URI 'http://%5Bredacted%5D#localhost:8998?#' is not supported by any registered client factories.
at org.apache.livy.LivyClientBuilder.build(LivyClientBuilder.java:155)
at Client.<init>(Client.java:17)
at Client.main(Client.java:25)
I found out client instance stays null in LivyClientBuilder class.
client = factory.createClient(uri, this.config);
factory is an instance of LivyClientFactory interface.
The only class which implements the interface is RSCClientFactory.
In RSCClientFactory we have this piece of code:
if (!"rsc".equals(uri.getScheme())) {
return null;
}
I've tried "rsc" instead of "http", this is the error:
2018-09-15 11:32:55 ERROR RSCClient:340 - RPC error.
java.util.concurrent.ExecutionException: javax.security.sasl.SaslException: Client closed before SASL negotiation finished.
javax.security.sasl.SaslException: Client closed before SASL negotiation finished.
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
at org.apache.livy.rsc.rpc.Rpc$SaslClientHandler.dispose(Rpc.java:419)
at org.apache.livy.rsc.JobHandleImpl.get(JobHandleImpl.java:60)
at org.apache.livy.rsc.rpc.SaslHandler.channelInactive(SaslHandler.java:92)
at Client.main(Client.java:39)
Apache Livy is running on http://localhost:8998 then I think we need submit our jar file to this address, but I don't understand "rsc" there.
I would appreciate if anyone guides me about these problems.
You just need to pass your URL as a String:
LivyClient client = new LivyClientBuilder()
.setURI(new URI("http://localhost:8998"))
.build();
After that you can add your *.jar file with:
client.addJar("file://...yourPathToJarHere.../*.jar");
or
client.uploadJar(new File("...."));
It depends on your cluster configuration. You can find full java API description here: https://livy.incubator.apache.org/docs/latest/api/java/index.html
The only class which implements the interface is RSCClientFactory.
Just add livy-client-http to your classpath.
https://github.com/apache/incubator-livy/blob/412ccc8fcf96854fedbe76af8e5a6fec2c542d25/client-http/src/main/java/org/apache/livy/client/http/HttpClientFactory.java#L29
https://github.com/apache/incubator-livy/blob/56c76bc2d4563593edce062a563603fe63e5a431/examples/src/main/java/org/apache/livy/examples/PiApp.java#L79
Docs: https://livy.incubator.apache.org/docs/latest/programmatic-api.html
Related
I am new to apache camel. What I am trying to do is I have exposed and Rest api to get data.
From that I need to communicate to an existing TCP server(Simple java server application) to retrieve data and send back to a client. What I have picked is Apache camel to do this integration.
rest()
.consumes("application/json").produces("application/json")
.get("/weather2/{city}").outType(WeatherDto.class).to("direct:get-weather-data")
from("direct:get-weather-data")
.process(this::setTCPMsg)
.to("netty://tcp://127.0.0.1:9898")
Above is the way I have defined the routes but when I try to run the appication I get below eror
org.apache.camel.FailedToCreateRouteException: Failed to create route route5 at: >>> To[netty://tcp://127.0.0.1:9898] <<< in route: Route(route5)[From[direct:get-weather-data] -> [process[Proc... because of No endpoint could be found for: netty://tcp://127.0.0.1:9898, please check your classpath contains the needed Camel component jar.
Please advice How to solve this issue.
We are running a simple application that connects to Firebase are reads some data. It fails to connect with the following timeout error:
#firebase/database: FIREBASE WARNING: {"code":"app/invalid-credential",
"message":"Credential implementation provided to initializeApp()
via the \"credential\" property failed to fetch a valid Google OAuth2 access token
with the following error: \"Failed to parse access token response: Error: Error
while making request: connect ETIMEDOUT
We are behind Firewall / Proxy and it appears that is blocking traffic to/from Firebase and hence failed connection. My question is what ports need to be opened and to what destination URLs to make this application work normally?
Any help will be much appreciated!
Finally, after struggling with the issue for several days got it working. Needed to contact network team and request to perform following actions:
Open ports 5228, 5229, 5230 for Firebase communication.
Opened communication at proxy level between the source server and following URLs:
fcm.googleapis.com
gcm-http.googleapis.com
accounts.google.com
{project-name}.firebaseio.com
Added following code in my node.js application:
var globalTunnel = require('global-tunnel-ng');
globalTunnel.initialize({
host: '<proxy-url>',
port: <proxy-port>,
//proxyAuth: 'userId:password', // optional authentication
sockets: 50 // optional pool size for each http and https
});
Installed module global-tunnel-ng:
npm install global-tunnel-ng
It solved the my problem and I hope it can help others too. :-)
I used Wireshark to monitor a local install of a Node.js application using the Admin SDK for firestore. I also referenced this list by Netify. This is what I found:
*.firebaseio.com
*.google.com
*.google-analytics.com
*.googleapis.com
*.firebase.com
*.firebaseapp.com
We are trying to deploy an EAR on WebSphere Liberty.
Our application contains an EJB-module, which contains and EJB that makes a call to another SOAP server.
The WSDL of the service defines a wsp:Policy with ExactlyOne of http:BasicAuthentication xmlns:http="http://schemas.microsoft.com/ws/06/2004/policy/http"/
After deployment when we send a request to our application, which would trigger that SOAP-call we get an error: None of the policy alternatives can be satisfied.
I found some java-code on how to solve this
HTTPConduit http = (HTTPConduit) client.getConduit();
http.getAuthorization().setUserName("user");
http.getAuthorization().setPassword("pass");
But I do not want to do this in the Java-code but I want to make it part of the server config.
I found several helpful links, but still could not get it working.
Does anybody have any suggestions on how I can set this up?
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_wssec_migrating.html
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/twlp_sec_ws_clientcert.html
You could use the JNDI feature to express the userid and password in server.xml, then have your java code pull it out of JNDI.
https://www.ibm.com/support/knowledgecenter/en/SSD28V_8.5.5/com.ibm.websphere.wlp.core.doc/ae/twlp_dep_jndi.html
I try to run a ASP.NET Core MVC Web application on the Swisscom Appcloud. But when I start the Application I get following Error-Message in the Console:
2017-01-24 14:29:53 [CELL/0] ERR Timed out after 1m0s: health check never passed.
Its looks like the Appcloud cannot check the Health of my Application. Do I need to install a Nuget-Package or something else to get this up and running?
Thanks for your effort
By default, Cloud Foundry makes a health check by trying to connect to the port which the application is exposing.
If your application is not exposing ANY port (e.g., it's not a web service with APIs and so on), then you should add the health-check-type attribute to none, as described here.
If after that you still get errors, then I suggest you to find where your application is listening to a given port. In Cloud Foundry you must listen to $PORT, which is a environment variable. You can check an example of that here.
As gsmachado has already mentioned, you must listen to a specific port.
The .NET Core buildpack configures the app web server automatically so you don’t have to handle this yourself. But you have to prepare your app in a way that allows the buildpack to deliver this information via the command line to your app.
The buildpack will start your app with the following command:
$ dotnet run --server.urls http://0.0.0.0:${PORT}
Therefore you have to add the command line as a configuration provider and then add the UseConfiguration extension to pass the configuration to the WebHostBuilder
e.g.:
var config = new ConfigurationBuilder()
.AddCommandLine(args)
.Build();
var host = new WebHostBuilder()
.UseKestrel()
.UseConfiguration(config)
.UseStartup<Startup>()
.Build();
I'm using Kryonet with Slick2d to make a java game.
It works fine when running as a java application, however when running as an applet I get the following error:
00:00 INFO: [kryonet] Server opened.
00:04 DEBUG: [kryonet] Port 9991/TCP connected to: /(ip):55801
00:04 DEBUG: [kryo] Write: RegisterTCP
00:04 INFO: [kryonet] Connection 1 connected: /(ip)
00:04 INFO: [SERVER] Someone has connected.
00:04 ERROR: [kryonet] Error reading TCP from connection: Connection 1
com.esotericsoftware.kryonet.KryoNetException: Error during deserialization.
at com.esotericsoftware.kryonet.TcpConnection.readObject(TcpConnection.java:141)
at com.esotericsoftware.kryonet.Server.update(Server.java:192)
at com.esotericsoftware.kryonet.Server.run(Server.java:350)
at java.lang.Thread.run(Unknown Source)
Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow.
at com.esotericsoftware.kryo.io.Input.require(Input.java:162)
at com.esotericsoftware.kryo.io.Input.readLong(Input.java:621)
at com.esotericsoftware.kryo.io.Input.readDouble(Input.java:745)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$DoubleSerializer.read(DefaultSerializers.java:141)
at com.esotericsoftware.kryo.serializers.DefaultSerializers$DoubleSerializer.read(DefaultSerializers.java:131)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:735)
at com.esotericsoftware.kryonet.KryoSerialization.read(KryoSerialization.java:57)
at com.esotericsoftware.kryonet.TcpConnection.readObject(TcpConnection.java:139)
... 3 more
00:04 INFO: [SERVER] Someone has disconnected.
00:04 INFO: [kryonet] Connection 1 disconnected.
The server is running locally as a runnable jar and the client applet in a HTML file locally aswell, which is running xampp to act as a web server.
I've tried different serializers, buffer sizes and sending just single String/Booleans etc, it just doesn't seem to like anything.
The client connects to the server perfectly fine, however when it comes to sending any packets, I get the above error, no matter what packet is sent.
Any help/advice would be really appreciated - I've been stumped on this for a while!
Thanks
I believe I have the same problem or at least similar one. I am using Kryonet for server and client. The client is an applet and when I run it trough Eclipse's Applet Viewer it works fine. When I run it trough a web server I get similar errors. Client and server connect, server receives client's packets, but the client gives an error wherever it tries any deserialization. I found that the applet permissions are to blame. If you change the permissions of the Applet Viewer (if you are using Eclipse) to be the same as of a web page, you will get the same errors. The advantage is that you can then debug the problem.
To change the permissions for Eclipse:
Go to your project folder \bin\ and open "java.policy.applet". Inside you should have:
grant {
permission java.security.AllPermission;
};
Change that to:
grant {
permission java.io.FilePermission "<<ALL FILES>>", "read, write, execute, delete";
permission java.net.SocketPermission "*", "accept, connect, listen, resolve";
permission java.util.PropertyPermission "*", "read, write";
permission java.lang.RuntimePermission "*";
permission java.awt.AWTPermission "showWindowWithoutWarningBanner";
};
With this change I had the same behavior for Applet Viewer as with an embedded applet. This is not a full solution, but can help in finding the cause of the problem.
Update:
I have found what is the problem in my case. The problem is in the FieldSerializer and the other serializers using it. When a class is registered, the FieldSerializer goes over it's fields and set's them all to be accessible. This operation is not allowed for an applet. The result is wrong registration and serialization/deserialization. I have found 2 workarounds:
1) Use of another serializer. The default one is a FieldSerializer and can be changed using
public void setDefaultSerializer (Class<? extends Serializer> serializer)
another option is to set the serializer when registering each class. Do not use serializers based on the FieldSerializer.
2) Try to fix the FieldSerializer. What I am doing is not fully correct, but it works in my case. We will make the FieldSerializer continue the registration if setting of the accessibility is causing an Exception. Another thing we need to do is set all fields of the classes we register to public. TO change the FieldSerializer you need the Kryo sources. Go to FieldSerializer.java, mething rebuildCachedFields(). You will find the following code there:
if (!field.isAccessible()) {
if (!setFieldsAsAccessible) continue;
try {
field.setAccessible(true);
} catch (AccessControlException ex) {
continue;
}
}
You need to change that to:
if (!field.isAccessible()) {
if (setFieldsAsAccessible)
try {
field.setAccessible(true);
} catch (AccessControlException ex) {
}
}
The other thing that needs to be changed is all of the registered classes to have only public fields.
I have similar problem in gradle build. May be you need just increase memory (either heap or PermSize) for the applet JVM