I have a main method to run a Flink job which is in last calling env.execute() method for trigger the flink job, while I was running test, it is running indefinitely, what I'm doing wrong.
#ClassRule
public static DockerComposeContainer environment =
new DockerComposeContainer(new File("docker-compose.yml"))
.withExposedService("zookeeper_1", 2181)
.withExposedService("kafka_1", 29092);
#ClassRule
public static MiniClusterResource flinkCluster =
new MiniClusterResource(
new MiniClusterResourceConfiguration.Builder()
.setNumberSlotsPerTaskManager(2)
.setNumberTaskManagers(1)
.build());
#Test
public void testMain() throws Exception {
// trigger job, it create flink env, run a map function on unbounded stream, and at last call execute method
StreamingJob.main(null);
// code execution not coming here
assert (4 == 4);
}
It is normal for streaming jobs to run forever, unless they have bounded inputs. For testing you want to make sure that your input isn't unbounded -- so, for example, if you are implementing an integration test that reads from Kafka, you will want to use a deserializer that eventually returns true from isEndOfStream.
I think it is because you are calling the static method of your StreamingJob. You have some how call a method that inside it has the:
public void execute() throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// env.YOUR_DATASTREAM_PIPELINE
env.execute();
}
then your integration test will be something like:
#Test
public void test() throws Exception {
StreamingJob streamJob = new StreamingJob();
streamJob.execute();
// execute your assertions
}
take a look at this example of this integration test for Flink.
Related
I'm trying to get a responsive JavaFX graphical interface while executing a cmd command.
The command I'm executing is the following.
youtube-dl.exe --audio-format mp3 --extract-audio https://www.youtube.com/watch?v=l2vy6pJSo9c
As you see this is a youtube-downloader that converts a youtube link to an mp3-file.
I want this to be executed in a second thread and not in the main FX thread.
I've solved this by implementing interface Callable in the class StartDownloadingThread.
#Override
public Process call() throws Exception {
Process p = null;
p = ExecuteCommand(localCPara1, localCPara2, localDirectory).start();
try {
Thread.sleep(30);
}catch (InterruptedException e){}
return p;
}
The method ExecuteCommand just returns a ProcessBuilder object.
I try to use Thread.sleep to make the program return to the main thread and thus making the application responsive. Unfortunately the program still freezes.
This is how the method call is called.
ExecutorService pool = Executors.newFixedThreadPool(2);
StartDownloadingThread callable = new StartDownloadingThread(parameter1, parameter2, directory);
Future future = pool.submit(callable);
Process p = (Process) future.get();
p.waitFor();
How do I make my GUI responsive using the interface Callable?
Using a executor to run a task just for you to use the get method of the Future that is returned when submitting the task does not actually free the original thread to continue with other tasks. Later you even use the waitFor method on the original thread, which is likely to take even more time than anything you do in your Callable.
For this purpose the Task class may be better suited, since it allows you to handle success/failure on the application thread using event handlers.
Also please make sure an ExecutorService is shut down after you're done submitting tasks.
Task<Void> task = new Task<Void>() {
#Override
protected Void call() throws Exception {
Process p = null;
p = ExecuteCommand(localCPara1, localCPara2, localDirectory).start();
// why are you even doing this?
try {
Thread.sleep(30);
}catch (InterruptedException e){}
// do the rest of the long running things
p.waitFor();
return null;
}
};
task.setOnSucceeded(event -> {
// modify ui to show success
});
task.setOnFailed(event -> {
// modify ui to show failure
});
ExecutorService pool = Executors.newFixedThreadPool(2);
pool.submit(task);
// add more tasks...
// shutdown the pool not keep the jvm alive because of the pool
pool.shutdown();
I'm trying to migrate my app to work with RxJava.
I already use Retrofit and therefore I'm trying to use a Retrofit interface which methods return Observables.
However I'm now having issues with coding tests against it, as I can't get the Observable to run on the main thread; I'm trying to use Scheduler.immediate() for it.
It seems that Retrofit doesn't allow to override it's behaviour, which makes totally sense for the real execution flow, but it makes testing very difficult.
As I've just started with RxJava + Retrofit I just hope I'm doing something wrong instead.
Below is what the code looks like:
#Test
public void shouldCompleteRequest() {
SomeRestRequest request = new SomeRestRequest(arg1, arg2);
TestSubscriber<SomeRestResponse> testSubscriber = new TestSubscriber<>();
new SomeRestCommand(mRestApi,
arg1, arg2
Schedulers.immediate(),
Schedulers.immediate(),
mMockEventBus).execute(request, testSubscriber);
testSubscriber.assertCompleted();
}
where
public void execute(T request, Observer<S> observer) {
getCommand(request)
.observeOn(mObserveOnScheduler) //The test injects Schedulers.immediate()
.subscribeOn(mSubscribeOnScheduler) //The test injects Schedulers.immediate()
.subscribe(observer);
}
,
#Override
protected Observable<SomeRestResponse> getCommand(SomeRestRequest request) {
return mRestApi.restCommand(arg1, arg2);
}
and
public interface RestApi {
#GET("/someEndPoint")
Observable<SomeRestResponse> restCommand(#Query("arg1") String arg1, #Query("arg2") String arg2);
}
If you modify your test to add testSubscriber.awaitTerminalEvent();, then your test will wait for the call to complete and hence the test will pass. You will still have to do an assertCompleted() as the terminal event can be either of successful completion or error.
#Test
public void shouldCompleteRequest() {
SomeRestRequest request = new SomeRestRequest(arg1, arg2);
TestSubscriber<SomeRestResponse> testSubscriber = new TestSubscriber<>();
new SomeRestCommand(mRestApi,
arg1, arg2
Schedulers.immediate(),
Schedulers.immediate(),
mMockEventBus).execute(request, testSubscriber);
testSubscriber.awaitTerminalEvent(); // add this line here
testSubscriber.assertCompleted();
}
I looked up the source code of Retrofit 1.9.0 and as per RxSupport class, the call is always executed in a separate thread provided by the httpExecutor. Hence using Schedulers.immediate() did not cause the call to happen in the main thread.
This is my scenario, i use webdriver with testNG for doing data driven test. I am observing that the data i am 'seeing' in web app which is provided by #dataprovider is missing some value. For exg if i have an array as {"1","2","3","4","5"}, i am getting these value in webdriver script using testNG #dataprovider, i am observing in the 'Web GUI' initially 2 might be displayed, then in the next iteration 5 is displayed then the test stop.
I am assuming that TestNG is not waiting for webdriver to complete the function or process.
Here is my sample code
#Test (dataProviderClass=MyDataProviders.class)
public class MyWebDriverClass{
#Test(dataProvider = "theProviderName")
public void providerHomeCreateuser(String arg1,String arg2)
{
<..input arg1, arg2 to text fields..>
}
}
I understand somewhere i need to put a Thread.wait(), could any body guide me on this.
Data provider method is as follows
public class MyDataProviders {
...
...
#DataProvider (name="theProviderName")
public static Object[][] getData() throws Exception
{
Object retObject[][]=getTableArray("src\\com\\abcd\\resource\\TestData.xls", 5, "MyTestData");
return retObject;
}
I am learning WF4 and got stuck at the following place. Please help.Thanks.
1) I have created a static method, MyMethod in a static class called Worker. Within this method I call Thread.Sleep(3000) and then print "MyMethod" called.
2) I then created an activity, DoWork (DoWork.xaml) which consists of a InvokeMethod (The target type is the Worker class in step 1 and MethodName = MyMethod).
3) In the main method, I call 2 methods called OutputSequence() and OutputParallel() which are as follows
private static void OutputSequence()
{
Sequence s = new Sequence() { Activities = new DoWork(), new DoWork() } };
WorkflowInvoker.Invoke(s);
}
private static void OutputParallel()
{
Parallel p = new Parallel() { Branches = new DoWork(), new DoWork() } };
WorkflowInvoker.Invoke(p);
}
The OutputSequence() is OK as it calls the target method twice (in sequence) but the parallel one seems to execute sequentially as well. I expected it to execute in parallel.
What am I missing.
The Parallel activity is not what you think it is - it allows you to wait for things in parallel not to execute CPU based code in parallel. The WF4 threading mode is that there is exactly one thread at a time active in the workflow.
If you put two delays in the parallel then both of those waits would occur in parallel as opposed to sequentially as they would in a sequence
The idea is you want to wait for a number of actions when you don;t know the order in which they will occur. Then the parallel activity is complete when all of its child branches have completed
Actually Parallel activity really executes all branches one-by-one and has nothing related to concurrent code execution, like two thread do.
But there is MS sample, that shows "true" concurrent execution for blocks inside of parallel activity. There is the AsyncCodeActivity in the .net 4 that allows to get concurrent execution of activities. Please check http://msdn.microsoft.com/en-us/library/ee358731(VS.100).aspx
Below you can find copy-pasted sample from link above:
public sealed class GenerateRandom : AsyncCodeActivity<int>
{
static Random r = new Random();
protected override IAsyncResult BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, object state)
{
// Create a delegate that references the method that implements
// the asynchronous work. Assign the delegate to the UserState,
// invoke the delegate, and return the resulting IAsyncResult.
Func<int> GetRandomDelegate = new Func<int>(GetRandom);
context.UserState = GetRandomDelegate;
return GetRandomDelegate.BeginInvoke(callback, state);
}
protected override int EndExecute(AsyncCodeActivityContext context, IAsyncResult result)
{
// Get the delegate from the UserState and call EndInvoke
Func<int> GetRandomDelegate = (Func<int>)context.UserState;
return (int)GetRandomDelegate.EndInvoke(result);
}
int GetRandom()
{
// This activity simulates taking a few moments
// to generate the random number. This code runs
// asynchronously with respect to the workflow thread.
Thread.Sleep(5000);
return r.Next(1, 101);
}
}
hope this will help for someone else
In my Singleton-EJB i start a TimerService every 2 minutes. When a client access the test method
sometimes the application runs into a deadlock. The problem is, the test method calls a asynchronous method inside the EJB (see Method determineABC). The deadlock happens when the scheduleMethod tries to create a single action timer and therefore tries to acquire a lock (because hte timer callback method is annotated with LOCK.WRITE). At the same time we are already in the determineABC Method which tries to invoke the asynchronous method asynchMethod. Maybe the call of ejbLocal.asynchMethod(...); also tries to acquire a lock. Anyway here i run into a deadlock, because the asynchronous method is never called. So what is the problem?
Here is a source code snippet:
#Singleton
#Startup
#TransactionManagement(TransactionManagementType.CONTAINER)
#TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED)
#ConcurrencyManagement(ConcurrencyManagementType.CONTAINER)
public class XEJB implements XEJBLocal {
#javax.annotation.Resource(name = "x/XEJB/TimeService")
private TimerService timerService;
#javax.annotation.Resource
private SessionContext ctx;
#Schedule(minute = "*/2", hour = "*", persistent = false)
#Lock(LockType.READ)
private void scheduleMethod() {
// Create Single Action Timer
timerService.createSingleActionTimer(new Date(), new TimerConfig(null, false));
}
#Timeout
#Lock(LockType.WRITE)
private void timer(Timer timer) {
// Do something
}
#Override
#Lock(LockType.READ)
public B test(...) {
return determineABC(...);
}
#Lock(LockType.READ)
private B determineABC(...) {
XEJBLocal ejb= (XEJBLocal) ctx.getBusinessObject(ctx.getInvokedBusinessInterface());
Future<ArrayList> result = null;
result = ejb.asynchMethod(...);
result.get(4, TimeUnit.MINUTES); // Sometimes runs into a DEADLOCK
...
}
#Asynchronous
#Override
#Lock(LockType.READ)
public Future<ArrayList> asynchMethod(...) {
...
return new AsyncResult<ArrayList>(abcList);
}
The Deadlock also happens when i only use the #Schedule Method and no TimerService...
The DeadLock also happens when i do not use a Future Object but void as return type of the asynchronous Method.
When the timeout Exception is thrown the deadlock is solved. When i annotate the timer method with #AccessTimeout(2000) and this time is up the asynchronous method is called and therefore the deadlock is also solved.
When i use Locktype.READ for the timer Method no Deadlock happens. But why? What does the asychronous method call?
READ locks have to wait for WRITE locks to finish before they start their work. When timer() is working all your other invokations, even to READ methods, are going to wait. Are you sure the timeout happens in result.get(4, TimeUnit.MINUTES);?
I think you may be have access timeouts in test() invokation, way before reaching result.get(4, TimeUnit.MINUTES);.