I'm trying to migrate my app to work with RxJava.
I already use Retrofit and therefore I'm trying to use a Retrofit interface which methods return Observables.
However I'm now having issues with coding tests against it, as I can't get the Observable to run on the main thread; I'm trying to use Scheduler.immediate() for it.
It seems that Retrofit doesn't allow to override it's behaviour, which makes totally sense for the real execution flow, but it makes testing very difficult.
As I've just started with RxJava + Retrofit I just hope I'm doing something wrong instead.
Below is what the code looks like:
#Test
public void shouldCompleteRequest() {
SomeRestRequest request = new SomeRestRequest(arg1, arg2);
TestSubscriber<SomeRestResponse> testSubscriber = new TestSubscriber<>();
new SomeRestCommand(mRestApi,
arg1, arg2
Schedulers.immediate(),
Schedulers.immediate(),
mMockEventBus).execute(request, testSubscriber);
testSubscriber.assertCompleted();
}
where
public void execute(T request, Observer<S> observer) {
getCommand(request)
.observeOn(mObserveOnScheduler) //The test injects Schedulers.immediate()
.subscribeOn(mSubscribeOnScheduler) //The test injects Schedulers.immediate()
.subscribe(observer);
}
,
#Override
protected Observable<SomeRestResponse> getCommand(SomeRestRequest request) {
return mRestApi.restCommand(arg1, arg2);
}
and
public interface RestApi {
#GET("/someEndPoint")
Observable<SomeRestResponse> restCommand(#Query("arg1") String arg1, #Query("arg2") String arg2);
}
If you modify your test to add testSubscriber.awaitTerminalEvent();, then your test will wait for the call to complete and hence the test will pass. You will still have to do an assertCompleted() as the terminal event can be either of successful completion or error.
#Test
public void shouldCompleteRequest() {
SomeRestRequest request = new SomeRestRequest(arg1, arg2);
TestSubscriber<SomeRestResponse> testSubscriber = new TestSubscriber<>();
new SomeRestCommand(mRestApi,
arg1, arg2
Schedulers.immediate(),
Schedulers.immediate(),
mMockEventBus).execute(request, testSubscriber);
testSubscriber.awaitTerminalEvent(); // add this line here
testSubscriber.assertCompleted();
}
I looked up the source code of Retrofit 1.9.0 and as per RxSupport class, the call is always executed in a separate thread provided by the httpExecutor. Hence using Schedulers.immediate() did not cause the call to happen in the main thread.
Related
I have a main method to run a Flink job which is in last calling env.execute() method for trigger the flink job, while I was running test, it is running indefinitely, what I'm doing wrong.
#ClassRule
public static DockerComposeContainer environment =
new DockerComposeContainer(new File("docker-compose.yml"))
.withExposedService("zookeeper_1", 2181)
.withExposedService("kafka_1", 29092);
#ClassRule
public static MiniClusterResource flinkCluster =
new MiniClusterResource(
new MiniClusterResourceConfiguration.Builder()
.setNumberSlotsPerTaskManager(2)
.setNumberTaskManagers(1)
.build());
#Test
public void testMain() throws Exception {
// trigger job, it create flink env, run a map function on unbounded stream, and at last call execute method
StreamingJob.main(null);
// code execution not coming here
assert (4 == 4);
}
It is normal for streaming jobs to run forever, unless they have bounded inputs. For testing you want to make sure that your input isn't unbounded -- so, for example, if you are implementing an integration test that reads from Kafka, you will want to use a deserializer that eventually returns true from isEndOfStream.
I think it is because you are calling the static method of your StreamingJob. You have some how call a method that inside it has the:
public void execute() throws Exception {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
// env.YOUR_DATASTREAM_PIPELINE
env.execute();
}
then your integration test will be something like:
#Test
public void test() throws Exception {
StreamingJob streamJob = new StreamingJob();
streamJob.execute();
// execute your assertions
}
take a look at this example of this integration test for Flink.
I could not find a definitive answer to whether it is safe to spawn threads within session-scoped JSF managed beans. The thread needs to call methods on the stateless EJB instance (that was dependency-injected to the managed bean).
The background is that we have a report that takes a long time to generate. This caused the HTTP request to time-out due to server settings we can't change. So the idea is to start a new thread and let it generate the report and to temporarily store it. In the meantime the JSF page shows a progress bar, polls the managed bean till the generation is complete and then makes a second request to download the stored report. This seems to work, but I would like to be sure what I'm doing is not a hack.
Check out EJB 3.1 #Asynchronous methods. This is exactly what they are for.
Small example that uses OpenEJB 4.0.0-SNAPSHOTs. Here we have a #Singleton bean with one method marked #Asynchronous. Every time that method is invoked by anyone, in this case your JSF managed bean, it will immediately return regardless of how long the method actually takes.
#Singleton
public class JobProcessor {
#Asynchronous
#Lock(READ)
#AccessTimeout(-1)
public Future<String> addJob(String jobName) {
// Pretend this job takes a while
doSomeHeavyLifting();
// Return our result
return new AsyncResult<String>(jobName);
}
private void doSomeHeavyLifting() {
try {
Thread.sleep(SECONDS.toMillis(10));
} catch (InterruptedException e) {
Thread.interrupted();
throw new IllegalStateException(e);
}
}
}
Here's a little testcase that invokes that #Asynchronous method several times in a row.
Each invocation returns a Future object that essentially starts out empty and will later have its value filled in by the container when the related method call actually completes.
import javax.ejb.embeddable.EJBContainer;
import javax.naming.Context;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
public class JobProcessorTest extends TestCase {
public void test() throws Exception {
final Context context = EJBContainer.createEJBContainer().getContext();
final JobProcessor processor = (JobProcessor) context.lookup("java:global/async-methods/JobProcessor");
final long start = System.nanoTime();
// Queue up a bunch of work
final Future<String> red = processor.addJob("red");
final Future<String> orange = processor.addJob("orange");
final Future<String> yellow = processor.addJob("yellow");
final Future<String> green = processor.addJob("green");
final Future<String> blue = processor.addJob("blue");
final Future<String> violet = processor.addJob("violet");
// Wait for the result -- 1 minute worth of work
assertEquals("blue", blue.get());
assertEquals("orange", orange.get());
assertEquals("green", green.get());
assertEquals("red", red.get());
assertEquals("yellow", yellow.get());
assertEquals("violet", violet.get());
// How long did it take?
final long total = TimeUnit.NANOSECONDS.toSeconds(System.nanoTime() - start);
// Execution should be around 9 - 21 seconds
assertTrue("" + total, total > 9);
assertTrue("" + total, total < 21);
}
}
Example source code
Under the covers what makes this work is:
The JobProcessor the caller sees is not actually an instance of JobProcessor. Rather it's a subclass or proxy that has all the methods overridden. Methods that are supposed to be asynchronous are handled differently.
Calls to an asynchronous method simply result in a Runnable being created that wraps the method and parameters you gave. This runnable is given to an Executor which is simply a work queue attached to a thread pool.
After adding the work to the queue, the proxied version of the method returns an implementation of Future that is linked to the Runnable which is now waiting on the queue.
When the Runnable finally executes the method on the real JobProcessor instance, it will take the return value and set it into the Future making it available to the caller.
Important to note that the AsyncResult object the JobProcessor returns is not the same Future object the caller is holding. It would have been neat if the real JobProcessor could just return String and the caller's version of JobProcessor could return Future<String>, but we didn't see any way to do that without adding more complexity. So the AsyncResult is a simple wrapper object. The container will pull the String out, throw the AsyncResult away, then put the String in the real Future that the caller is holding.
To get progress along the way, simply pass a thread-safe object like AtomicInteger to the #Asynchronous method and have the bean code periodically update it with the percent complete.
Introduction
Spawning threads from within a session scoped managed bean is not necessarily a hack as long as it does the job you want. But spawning threads at its own needs to be done with extreme care. The code should not be written that way that a single user can for example spawn an unlimited amount of threads per session and/or that the threads continue running even after the session get destroyed. It would blow up your application sooner or later.
The code needs to be written that way that you can ensure that an user can for example never spawn more than one background thread per session and that the thread is guaranteed to get interrupted whenever the session get destroyed. For multiple tasks within a session you need to queue the tasks.
Also, all those threads should preferably be served by a common thread pool so that you can put a limit on the total amount of spawned threads at application level.
Managing threads is thus a very delicate task. That's why you'd better use the built-in facilities rather than homegrowing your own with new Thread() and friends. The average Java EE application server offers a container managed thread pool which you can utilize via among others EJB's #Asynchronous and #Schedule. To be container independent (read: Tomcat-friendly), you can also use the Java 1.5's Util Concurrent ExecutorService and ScheduledExecutorService for this.
Below examples assume Java EE 6+ with EJB.
Fire and forget a task on form submit
#Named
#RequestScoped // Or #ViewScoped
public class Bean {
#EJB
private SomeService someService;
public void submit() {
someService.asyncTask();
// ... (this code will immediately continue without waiting)
}
}
#Stateless
public class SomeService {
#Asynchronous
public void asyncTask() {
// ...
}
}
Asynchronously fetch the model on page load
#Named
#RequestScoped // Or #ViewScoped
public class Bean {
private Future<List<Entity>> asyncEntities;
#EJB
private EntityService entityService;
#PostConstruct
public void init() {
asyncEntities = entityService.asyncList();
// ... (this code will immediately continue without waiting)
}
public List<Entity> getEntities() {
try {
return asyncEntities.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new FacesException(e);
} catch (ExecutionException e) {
throw new FacesException(e);
}
}
}
#Stateless
public class EntityService {
#PersistenceContext
private EntityManager entityManager;
#Asynchronous
public Future<List<Entity>> asyncList() {
List<Entity> entities = entityManager
.createQuery("SELECT e FROM Entity e", Entity.class)
.getResultList();
return new AsyncResult<>(entities);
}
}
In case you're using JSF utility library OmniFaces, this could be done even faster if you annotate the managed bean with #Eager.
Schedule background jobs on application start
#Singleton
public class BackgroundJobManager {
#Schedule(hour="0", minute="0", second="0", persistent=false)
public void someDailyJob() {
// ... (runs every start of day)
}
#Schedule(hour="*/1", minute="0", second="0", persistent=false)
public void someHourlyJob() {
// ... (runs every hour of day)
}
#Schedule(hour="*", minute="*/15", second="0", persistent=false)
public void someQuarterlyJob() {
// ... (runs every 15th minute of hour)
}
#Schedule(hour="*", minute="*", second="*/30", persistent=false)
public void someHalfminutelyJob() {
// ... (runs every 30th second of minute)
}
}
Continuously update application wide model in background
#Named
#RequestScoped // Or #ViewScoped
public class Bean {
#EJB
private SomeTop100Manager someTop100Manager;
public List<Some> getSomeTop100() {
return someTop100Manager.list();
}
}
#Singleton
#ConcurrencyManagement(BEAN)
public class SomeTop100Manager {
#PersistenceContext
private EntityManager entityManager;
private List<Some> top100;
#PostConstruct
#Schedule(hour="*", minute="*/1", second="0", persistent=false)
public void load() {
top100 = entityManager
.createNamedQuery("Some.top100", Some.class)
.getResultList();
}
public List<Some> list() {
return top100;
}
}
See also:
Spawning threads in a JSF managed bean for scheduled tasks using a timer
I tried this and works great from my JSF managed bean
ExecutorService executor = Executors.newFixedThreadPool(1);
#EJB
private IMaterialSvc materialSvc;
private void updateMaterial(Material material, String status, Location position) {
executor.execute(new Runnable() {
public void run() {
synchronized (position) {
// TODO update material in audit? do we need materials in audit?
int index = position.getMaterials().indexOf(material);
Material m = materialSvc.getById(material.getId());
m.setStatus(status);
m = materialSvc.update(m);
if (index != -1) {
position.getMaterials().set(index, m);
}
}
}
});
}
#PreDestroy
public void destory() {
executor.shutdown();
}
I'm trying to get a responsive JavaFX graphical interface while executing a cmd command.
The command I'm executing is the following.
youtube-dl.exe --audio-format mp3 --extract-audio https://www.youtube.com/watch?v=l2vy6pJSo9c
As you see this is a youtube-downloader that converts a youtube link to an mp3-file.
I want this to be executed in a second thread and not in the main FX thread.
I've solved this by implementing interface Callable in the class StartDownloadingThread.
#Override
public Process call() throws Exception {
Process p = null;
p = ExecuteCommand(localCPara1, localCPara2, localDirectory).start();
try {
Thread.sleep(30);
}catch (InterruptedException e){}
return p;
}
The method ExecuteCommand just returns a ProcessBuilder object.
I try to use Thread.sleep to make the program return to the main thread and thus making the application responsive. Unfortunately the program still freezes.
This is how the method call is called.
ExecutorService pool = Executors.newFixedThreadPool(2);
StartDownloadingThread callable = new StartDownloadingThread(parameter1, parameter2, directory);
Future future = pool.submit(callable);
Process p = (Process) future.get();
p.waitFor();
How do I make my GUI responsive using the interface Callable?
Using a executor to run a task just for you to use the get method of the Future that is returned when submitting the task does not actually free the original thread to continue with other tasks. Later you even use the waitFor method on the original thread, which is likely to take even more time than anything you do in your Callable.
For this purpose the Task class may be better suited, since it allows you to handle success/failure on the application thread using event handlers.
Also please make sure an ExecutorService is shut down after you're done submitting tasks.
Task<Void> task = new Task<Void>() {
#Override
protected Void call() throws Exception {
Process p = null;
p = ExecuteCommand(localCPara1, localCPara2, localDirectory).start();
// why are you even doing this?
try {
Thread.sleep(30);
}catch (InterruptedException e){}
// do the rest of the long running things
p.waitFor();
return null;
}
};
task.setOnSucceeded(event -> {
// modify ui to show success
});
task.setOnFailed(event -> {
// modify ui to show failure
});
ExecutorService pool = Executors.newFixedThreadPool(2);
pool.submit(task);
// add more tasks...
// shutdown the pool not keep the jvm alive because of the pool
pool.shutdown();
We basically have a class that looks like this below that is using the Castle.DynamicProxy for Interception.
using System;
using System.Collections.Concurrent;
using System.Reflection;
using System.Threading;
using System.Threading.Tasks;
using Castle.DynamicProxy;
namespace SaaS.Core.IoC
{
public abstract class AsyncInterceptor : IInterceptor
{
private readonly ILog _logger;
private readonly ConcurrentDictionary<Type, Func<Task, IInvocation, Task>> wrapperCreators =
new ConcurrentDictionary<Type, Func<Task, IInvocation, Task>>();
protected AsyncInterceptor(ILog logger)
{
_logger = logger;
}
void IInterceptor.Intercept(IInvocation invocation)
{
if (!typeof(Task).IsAssignableFrom(invocation.Method.ReturnType))
{
InterceptSync(invocation);
return;
}
try
{
CheckCurrentSyncronizationContext();
var method = invocation.Method;
if ((method != null) && typeof(Task).IsAssignableFrom(method.ReturnType))
{
var taskWrapper = GetWrapperCreator(method.ReturnType);
Task.Factory.StartNew(
async () => { await InterceptAsync(invocation, taskWrapper).ConfigureAwait(true); }
, // this will use current synchronization context
CancellationToken.None,
TaskCreationOptions.AttachedToParent,
TaskScheduler.FromCurrentSynchronizationContext()).Wait();
}
}
catch (Exception ex)
{
//this is not really burring the exception
//excepiton is going back in the invocation.ReturnValue which
//is a Task that failed. with the same excpetion
//as ex.
}
}
....
Initially this code was:
Task.Run(async () => { await InterceptAsync(invocation, taskWrapper)).Wait()
But we were losing HttpContext after any call to this, so we had to switch it to:
Task.Factory.StartNew
So we could pass in the TaskScheduler.FromCurrentSynchronizationContext()
All of this is bad because we are really just swapping one thread for another thread. I would really love to change the signature of
void IInterceptor.Intercept(IInvocation invocation)
to
async Task IInterceptor.Intercept(IInvocation invocation)
And get rid of the Task.Run or Task.Factory and just make it:
await InterceptAsync(invocation, taskWrapper);
The problem is Castle.DynamicProxy IInterecptor won't allow this. I really want do an await in the Intercept. I could do .Result but then what is the point of the async call I am calling? Without being able to do the await I lose out of the benefit of it being able to yield this threads execution. I am not stuck with Castle Windsor for their DynamicProxy so I am looking for another way to do this. We have looked into Unity, but I don't want to replace our entire AutoFac implementation.
Any help would be appreciated.
All of this is bad because we are really just swapping one thread for another thread.
True. Also because the StartNew version isn't actually waiting for the method to complete; it will only wait until the first await. But if you add an Unwrap() to make it wait for the complete method, then I strongly suspect you'll end up with a deadlock.
The problem is Castle.DynamicProxy IInterecptor won't allow this.
IInterceptor does have a design limitation that it must proceed synchronously. So this limits your interception capabilities: you can inject synchronous code before or after the asynchronous method, and asynchronous code after the asynchronous method. There's no way to inject asynchronous code before the asynchronous method. It's just a limitation of DynamicProxy, one that would be extremely painful to correct (as in, break all existing user code).
To do the kinds of injection that is supported, you have to change your thinking a bit. One of the valid mental models of async is that a Task returned from a method represents the execution of that method. So, to append code to that method, you would call the method directly and then replace the task return value with an augmented one.
So, something like this (for return types of Task):
protected abstract void PreIntercept(); // must be sync
protected abstract Task PostInterceptAsync(); // may be sync or async
// This method will complete when PostInterceptAsync completes.
private async Task InterceptAsync(Task originalTask)
{
// Asynchronously wait for the original task to complete
await originalTask;
// Asynchronous post-execution
await PostInterceptAsync();
}
public void Intercept(IInvocation invocation)
{
// Run the pre-interception code.
PreIntercept();
// *Start* the intercepted asynchronous method.
invocation.Proceed();
// Replace the return value so that it only completes when the post-interception code is complete.
invocation.ReturnValue = InterceptAsync((Task)invocation.ReturnValue);
}
Note that the PreIntercept, the intercepted method, and PostInterceptAsync are all run in the original (ASP.NET) context.
P.S. A quick Google search for async DynamicProxy resulted in this. I don't have any idea how stable it is, though.
I m sending several retrofit calls via SyncAdapter onPerformSync and I m trying to regulate http calls by sending out via a try/catch sleep statement. However, this is blocking the UI and will be not responsive only after all calls are done.
What is a better way to regulate network calls (with a sleep timer) in background in onPerformSync without blocking UI?
#Override
public void onPerformSync(Account account, Bundle extras, String authority, ContentProviderClient provider, SyncResult syncResult) {
String baseUrl = BuildConfig.API_BASE_URL;
Retrofit retrofit = new Retrofit.Builder()
.baseUrl(baseUrl)
.addConverterFactory(GsonConverterFactory.create())
.build();
service = retrofit.create(HTTPService.class);
Call<RetroFitModel> RetroFitModelCall = service.getRetroFit(apiKey, sortOrder);
RetroFitModelCall.enqueue(new Callback<RetroFitModel>() {
#Override
public void onResponse(Response<RetroFitModel> response) {
if (!response.isSuccess()) {
} else {
List<RetroFitResult> retrofitResultList = response.body().getResults();
Utility.storeList(getContext(), retrofitResultList);
for (final RetroFitResult result : retrofitResultList) {
RetroFitReview(result.getId(), service);
try {
// Sleep for SLEEP_TIME before running RetroFitReports & RetroFitTime
Thread.sleep(SLEEP_TIME);
} catch (InterruptedException e) {
}
RetroFitReports(result.getId(), service);
RetroFitTime(result.getId(), service);
}
}
}
#Override
public void onFailure(Throwable t) {
Log.e(LOG_TAG, "Error: " + t.getMessage());
}
});
}
}
The "onPerformSync" code is executed within the "SyncAdapterThread" thread, not within the Main UI thread. However this could change when making asynchronous calls with callbacks (which is our case here).
Here you are using an asynchronous call of the Retrofit "call.enqueue" method, and this has an impact on thread execution. The question we need to ask at this point:
Where callback methods are going to be executed?
To get the answer to this question, we have to determine which Looper is going to be used by the Handler that will post callbacks.
In case we are playing with handlers ourselves, we can define the looper, the handler and how to process messages/runnables between handlers. But this time it is different because we are using a third party framework (Retrofit). So we have to know which looper used by Retrofit?
Please note that if Retrofit didn't already define his looper, you
could have caught an exception saying that you need a looper to
process callbacks. In other words, an asynchronous call needs to be in
a looper thread in order to post callbacks back to the thread from
where it was executed.
According to the code source of Retrofit (Platform.java):
static class Android extends Platform {
#Override CallAdapter.Factory defaultCallAdapterFactory(Executor callbackExecutor) {
if (callbackExecutor == null) {
callbackExecutor = new MainThreadExecutor();
}
return new ExecutorCallAdapterFactory(callbackExecutor);
}
static class MainThreadExecutor implements Executor {
private final Handler handler = new Handler(Looper.getMainLooper());
#Override public void execute(Runnable r) {
handler.post(r);
}
}
}
You can notice "Looper.getMainLooper()", which means that Retrofit will post messages/runnables into the main thread message queue (you can do research on this for further detailed explanation). Thus the posted message/runnable will be handled by the main thread.
So that being said, the onResponse/onFailure callbacks will be executed in the main thread. And it's going to block the UI, if you are doing too much work (Thread.sleep(SLEEP_TIME);). You can check it by yourself: just make a breakpoint in "onResponse" callback and check in which thread it is running.
So how to handle this situation? (the answer to your question about Retrofit use)
Since we are already in a background thread (SyncAdapterThread), so there is no need to make asynchronous calls in your case. Just make a Retrofit synchronous call and then process the result, or log a failure. This way, you will not block the UI.