I am facing a problem in modifying an object in a loop in the same thread and the modification seems to work fine if the modification is done in different threads.
Here is the code that I have written for the same.
The parent method call is:
public void task(){
List<URLTask> tasks = dService.getTasksForStatusReport(System.currentTimeMillis());
System.out.println("scheduler called to send mail");
this.sendMail(tasks,false);
}
service to fetch the list of the tasks:
public List<URLTask> getTasksForStatusReport(long timestamp) {
PersistenceManager pm = pmf.getPersistenceManager();
Query q = pm.newQuery("SELECT FROM " + URLTask.class.getName() + " WHERE nextMailTimestamp <= "+timestamp+" && isDeleted != true");
List<URLTask> c = (List<URLTask>)q.execute();
pm.detachCopyAll(c);//I have tried commenting and uncommenting this line but no effect
return c;
}
The code to for the sendMail(tasks,boolean) is as follow:
public void sendMail(List<URLTask> tasks , boolean consolidatedEmail){
for(URLTask task: tasks){
task.setNextMailTimestamp(System.currentTimeMillis() + task.getMailIntervalInMilliseconds());
dService.saveUrlTask(task);
}
}
}
saveUrlTask(task) method is:
public URLTask saveUrlTask(URLTask task) {
PersistenceManager pm = pmf.getPersistenceManager();
Transaction tx = pm.currentTransaction();
URLTask taskCopy;
tx.begin();
Query q = pm.newQuery("SELECT FROM " + URLTask.class.getName() +
" WHERE isDeleted != 0 && id == \"" + task.getId() + "\"");
long bfr = System.currentTimeMillis();
List<URLTask> taskList = (List<URLTask>)q.execute();
long aft = System.currentTimeMillis();
System.out.println("getting url task time = "+ (aft-bfr));
URLTask oldTask = taskList.get(0);
oldTask.setRepeatIntervalInSeconds(task.getRepeatIntervalInSeconds());
oldTask.setUrl(task.getUrl());
oldTask.setDeleted(task.isDeleted());
oldTask.setNextMailTimestamp(task.getNextMailTimestamp());
oldTask.setMailIntervalInMilliseconds(task.getMailIntervalInMilliseconds());
oldTask.setUserDefinedErrorMessages(task.getUserDefinedErrorMessages());
oldTask.setUpdateTimeStamp(System.currentTimeMillis());
oldTask.setMailIntervalInMilliseconds(task.getMailIntervalInMilliseconds());
oldTask.setNextMailTimestamp(task.getNextMailTimestamp());
oldTask.setUserDefinedErrorMessages(task.getUserDefinedErrorMessages());
bfr = System.currentTimeMillis();
pm.makePersistent(oldTask);
aft = System.currentTimeMillis();
System.out.println("updating url task time = "+ (aft-bfr));
taskCopy = (URLTask)(pm.detachCopy(oldTask));
tx.commit();
pm.close();
return taskCopy;
}
The code works perfectly when the size of the list is 1 and as the size is 2 it again works fine for the first one but throws the exception when runs for the second time.
For reference the exception is:
Exception in thread "HBase Connection Evictor" java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1143)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1241)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:826)
at org.apache.hadoop.hbase.client.HTable.close(HTable.java:838)
at org.datanucleus.store.hbase.HBaseManagedConnection.closeTables(HBaseManagedConnection.java:169)
at org.datanucleus.store.hbase.HBaseManagedConnection.dispose(HBaseManagedConnection.java:154)
at org.datanucleus.store.hbase.HBaseConnectionPool.disposeTimedOutConnections(HBaseConnectionPool.java:94)
at org.datanucleus.store.hbase.HBaseConnectionPool.access$000(HBaseConnectionPool.java:27)
at org.datanucleus.store.hbase.HBaseConnectionPool$1.run(HBaseConnectionPool.java:105)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Any form of help would be highly appreciable.
Thanks in advance!!
devsri
Related
I have a custom AX service operation that can take 5+ minutes to complete and I'm trying to figure out how to abort it from my .NET application, but aborting the client doesn't seem to do anything?
The problem is if I call the operation and the service times out, the AX operation continues on until completion, so I lose visibility to the results (success/failure). An example being a long-running posting operation where I don't know if it posted successfully or not.
I've created a simple demo app where I can't seem to get the operation to abort. In the below code I just create a transaction (ttsbegin/ttscommit), insert into a table at start, sleep, insert into table at end.
Sample AX X++ Service Code:
[SysEntryPointAttribute(true)]
public str callAXTimeDelay(int _sleepSeconds)
{
Table1 table1;
ttsBegin;
table1.clear();
table1.SleepData = strFmt("STARTED: %1", DateTimeUtil::utcNow());
table1.insert();
ttsCommit;
sleep(_sleepSeconds * 1000);
ttsBegin;
table1.clear();
table1.SleepData = strFmt("COMPLETED: %1", DateTimeUtil::utcNow());
table1.insert();
ttsCommit;
return strFmt("COMPLETED: %1", DateTimeUtil::utcNow());
}
Then when I call it from .NET, the abort doesn't seem to work? Start/Complete records are still inserted into table1 even though the abort is called before the 15 seconds have completed?
Sample .NET code:
internal class Program
{
static void Main(string[] args)
{
new Program().Run();
Console.WriteLine("Ended, press any key to exit...");
Console.ReadKey();
}
public void Run()
{
AXServicesClient axClient15Sec = new AXServicesClient();
AXServicesClient axClient5sec = new AXServicesClient();
var job15sec = DoLongRunningCall(axClient15Sec, 15);
var job5sec = DoLongRunningCall(axClient5sec, 5);
try
{
var result = Task.Run(() => Task.WhenAny(job15sec, job5sec)).GetAwaiter().GetResult();
if (result == job15sec)
{
Console.WriteLine("job15sec finished first, aborting job5sec");
axClient5sec.Abort();
}
else if (result == job5sec)
{
// This code gets executed because the 5 second job completed and
// it tries to cancel the 15-sec job, but the table ends up with data!
Console.WriteLine("job5sec finished first, aborting job15sec");
axClient15Sec.Abort();
}
}
catch (Exception e)
{
Console.WriteLine("Exception: " + e.Message);
axClient15Sec.Abort();
axClient5sec.Abort();
}
axClient15Sec.Close();
axClient5sec.Close();
}
public async Task<string> DoLongRunningCall(AXServicesClient client, int seconds)
{
var result = await client.callAXTimeDelay(new CallContext
{
Company = "ABCD",
Language = "en-us"
}, seconds);
return result.response;
}
}
While trying the OSGi PushStream library I felt that it was really slow. I have created two methods that do the same thing one using the PushStream and the other a simple BlockingQueue (see code below), the result is the following:
Queue needs 3 milliseconds to process 1000 events.
PushStream needs 31331 milliseconds to process 1000 events.
Why the PushStream is slower? What I am doing wrong?
Code
With PushStream:
public class TestPush{
#Test
public void testPushStream() throws Exception {
final PromiseFactory pf = new PromiseFactory(PromiseFactory.inlineExecutor());
final PushStreamProvider psp = new PushStreamProvider();
final SimplePushEventSource<Integer> source =
psp.buildSimpleEventSource(Integer.class).withQueuePolicy(QueuePolicyOption.BLOCK).build();
final Deferred<Instant> startD = pf.deferred();
final Deferred<Instant> endD = pf.deferred();
psp.createStream(source).onClose(() -> endD.resolve( Instant.now()) ).forEach((i) -> {
if (i == 0) {
startD.resolve( Instant.now() );
}
});
final Promise<Long> nbEvent = psp.createStream(source).count();
for (int i = 0; i < 1000; i++) {
source.publish(i);
}
source.endOfStream();
System.out.println("PushStream needs "
+ Duration.between( startD.getPromise().getValue(), endD.getPromise().getValue() ).toMillis()
+ " milliseconds to process " + nbEvent.getValue() + " events.");
}
With ArrayBlockingQueue:
#Test
public void testBlockingQueue() throws Exception {
final PromiseFactory pf = new PromiseFactory(PromiseFactory.inlineExecutor());
final Executor e = Executors.newFixedThreadPool(1);
final ArrayBlockingQueue<Integer> abq = new ArrayBlockingQueue<>(32);
final Deferred<Instant> startD = pf.deferred();
final Deferred<Instant> endD = pf.deferred();
final Deferred<Integer> nbEvent = pf.deferred();
e.execute( () -> {
try {
Integer i = 0;
Integer last = 0;
do {
i = abq.take();
if (i == 0) {
startD.resolve(Instant.now());
} else if (i != -1) {
last = i;
}
}
while (i != -1);
endD.resolve(Instant.now());
nbEvent.resolve(last + 1);
}
catch (final InterruptedException exception) {
exception.printStackTrace();
}
});
for (int i = 0; i < 1000; i++) {
abq.put(i);
}
abq.put(-1);
System.out.println("Queue needs "
+ Duration.between( startD.getPromise().getValue(), endD.getPromise().getValue() ).toMillis()
+ " milliseconds to process " + nbEvent.getPromise().getValue() + " events.");
}
}
This is a fun question :)
Why the PushStream is slower? What I am doing wrong?
Thank you for not just assuming that the PushStream implementation sucks. In this case it is slower because (probably without realising) you asked it to be!
Part 1 - Buffering
By default PushStreams are buffered. This means that they include a queue into which events are placed before they are processed. Buffering therefore does a few things which negatively affect the speed of throughput.
It adds an extra queue/dequeue step into the pipeline
It adds an extra thread switch in the event processing
The default policy for a buffer is to return back pressure related to how full the buffer is.
In this case the vast majority of the slowdown is because of the back pressure. When you create a stream using psp.createStream(source) it is set up with a buffer of 32 elements and a linear back pressure policy based on the size of the buffer, returning one second when full, and 31 millis when it has one item in it. It is worth noting that 31 millis per element adds up to 30 seconds!
Importantly, the SimplePushEventSource always honours back pressure requests from the consumers that are added to it. This means that you may be pumping events into the SimplePushEventSource as fast as you can, but they will only be delivered as fast as they are requested by the pipeline.
If we remove the buffering from the push streams that you are creating then we get the following test:
#Test
public void testPushStream2() throws Exception {
final PromiseFactory pf = new PromiseFactory(PromiseFactory.inlineExecutor());
final PushStreamProvider psp = new PushStreamProvider();
final SimplePushEventSource<Integer> source =
psp.buildSimpleEventSource(Integer.class)
.withQueuePolicy(QueuePolicyOption.BLOCK)
.build();
final Deferred<Instant> startD = pf.deferred();
final Deferred<Instant> endD = pf.deferred();
psp.buildStream(source).unbuffered().build().onClose(() -> endD.resolve( Instant.now()) ).forEach((i) -> {
if (i == 0) {
startD.resolve( Instant.now() );
}
});
final Promise<Long> nbEvent = psp.buildStream(source).unbuffered().build().count();
for (int i = 0; i < 1000; i++) {
source.publish(i);
}
source.endOfStream();
System.out.println("PushStream needs "
+ Duration.between( startD.getPromise().getValue(), endD.getPromise().getValue() ).toMillis()
+ " milliseconds to process " + nbEvent.getValue() + " events.");
}
The result of running this (on my machine) is:
PushStream needs 39 milliseconds to process 1000 events.
This is obviously much closer to what you would expect, but it is still noticeably slower. Note that we could have still had some buffering, but tuned the PushbackPolicy. This would have given us faster throughput, but not quite as fast as this.
Part 2 - Pipeline lengths
The next thing to notice is that you are using an onClose() handler. This adds an extra stage into your push stream pipeline. You can actually move the onClose to be a result of the promise, decreasing the length of your pipeline (you only need to run it once).
#Test
public void testPushStream3() throws Exception {
final PromiseFactory pf = new PromiseFactory(PromiseFactory.inlineExecutor());
final PushStreamProvider psp = new PushStreamProvider();
final SimplePushEventSource<Integer> source =
psp.buildSimpleEventSource(Integer.class)
.withQueuePolicy(QueuePolicyOption.BLOCK)
.build();
final Deferred<Instant> startD = pf.deferred();
final Deferred<Instant> endD = pf.deferred();
psp.buildStream(source).unbuffered().build().forEach((i) -> {
if (i == 0) {
startD.resolve( Instant.now() );
}
});
final Promise<Long> nbEvent = psp.buildStream(source).unbuffered().build().count()
.onResolve(() -> endD.resolve( Instant.now()));
for (int i = 0; i < 1000; i++) {
source.publish(i);
}
source.endOfStream();
System.out.println("PushStream needs "
+ Duration.between( startD.getPromise().getValue(), endD.getPromise().getValue() ).toMillis()
+ " milliseconds to process " + nbEvent.getValue() + " events.");
}
The result of this version (on my machine) is:
PushStream needs 21 milliseconds to process 1000 events.
Part 3 - Multiplexing delivery
A key difference between the "raw array blocking queue" example and the PushStream example is that you actually create two PushStreams. The first does the work to capture the start time, the second to count the events. This forces the SimplePushEventSource to multiplex the events over multiple consumers.
What if we collapsed the behaviour into a single pipeline so that the SimplePushEventSource could use a fast-path delivery?
#Test
public void testPushStream4() throws Exception {
final PromiseFactory pf = new PromiseFactory(PromiseFactory.inlineExecutor());
final PushStreamProvider psp = new PushStreamProvider();
final SimplePushEventSource<Integer> source =
psp.buildSimpleEventSource(Integer.class)
.withQueuePolicy(QueuePolicyOption.BLOCK)
.build();
final Deferred<Instant> startD = pf.deferred();
final Deferred<Instant> endD = pf.deferred();
final Promise<Long> nbEvent = psp.buildStream(source).unbuffered().build()
.filter(i -> {
if (i == 0) {
startD.resolve( Instant.now() );
}
return true;
})
.count()
.onResolve(() -> endD.resolve( Instant.now()));
for (int i = 0; i < 1000; i++) {
source.publish(i);
}
source.endOfStream();
System.out.println("PushStream needs "
+ Duration.between( startD.getPromise().getValue(), endD.getPromise().getValue() ).toMillis()
+ " milliseconds to process " + nbEvent.getValue() + " events.");
}
The result of this version (on my machine) is:
PushStream needs 3 milliseconds to process 1000 events.
Summary
PushStreams are a fast, effective way to consume asynchronously arriving events, but it is very important to understand about what buffering behaviour is suitable for your application. If you have a big lump of data that you want to iterate over very quickly then you need to be careful how you set things up, as the buffering defaults are designed for a different use case!
Hello there I am trying to force a promise to end to get the result from it but it just stucks on loading.
public class CacheController extends Controller {
private AsyncCacheApi cache;
public Result cache()
{
String test = "nice";
cache.set("item.key", test, 15);
Customer user = new Customer("Ana", 12);
CompletionStage<Done> result = cache.set(user.getName(), user);
block(result);
return ok("Cached");
}
public Result checkCache() throws Exception
{
Logger.info("start");
//CompletionStage<String> news = cache.get("item.key");
//news.thenRun(() -> System.out.println("works"));
CompletionStage<Customer> result = cache.get("Ana");
Logger.info("step 1");
Logger.info(cache.get("Ana").toString());
Logger.info("Step 2");
Customer c = block(result);
Logger.info("Step 3 " + c.getName());
//result.thenRun(() -> setUser(result)).thenRun(() -> Logger.info(user.getName() + " " + user.getAge()));
return ok("cancan");
}
private <T> T block(CompletionStage<T> stage) {
try {
return stage.toCompletableFuture().get();
} catch (Throwable e) {
throw new RuntimeException(e);
}
}
}
When trying to load the page it gets stuck after step2 at line 56: Customer c = block(result); by my guesses
Any ideas to fix it?
#Codrin
I had the same problem. But, see https://www.playframework.com/documentation/2.6.x/JavaCache#Setting-the-execution-context
By default, all Ehcache operations are blocking, and async implementations will block threads in the default execution context.
Maybe CompletableFuture.get() gets stuck because it is executed in the same thread with the caller.
Referring to the linked page, I added snippet below to my application.conf and it worked.
play.cache.dispatcher = "contexts.blockingCacheDispatcher"
contexts {
blockingCacheDispatcher {
fork-join-executor {
parallelism-factor = 3.0
}
}
}
I'm trying to make use of Java signatures in a pretty time sensitive setting.
I've come across some interesting behavior when signing data of a few hundred bytes. I am reusing a generated key, but recreating the Signature object each time I sign. The first time the signing happens it takes from anywhere from 50-100ms depending on the machine being used. However, any subsequent times a signature is computed over new data (using the same key) the time is reduced down to 1-2ms. I'm using SHA512 with RSA so I expected it to be heavier.
Can anyone explain why this happens? A test class I am using is pasted below (which is using identical code to my target applictaion).
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.PrivateKey;
import java.security.PublicKey;
import java.security.SecureRandom;
import java.security.Security;
import java.security.Signature;
public class MainClass {
public static void main(String args[]) throws Exception {
//Generate key pair
long start = System.currentTimeMillis();
KeyPair keyPair = generateKeyPair(999);
long end = System.currentTimeMillis();
System.out.println("KeyGen: " + (end - start));
//Sign first piece of data
byte[] data = { ** a few hundred bytes** };
start = System.currentTimeMillis();
byte[] digitalSignature = signData(data, keyPair.getPrivate());
end = System.currentTimeMillis();
System.out.println("Sign: " + (end - start));
boolean verified;
byte[] data2 = {** a different few hundred bytes ** };
//sign second piece of data
start = System.currentTimeMillis();
digitalSignature = signData(data2, keyPair.getPrivate());
end = System.currentTimeMillis();
System.out.println(" Second Sign: " + (end - start));
//verify second signature
start = System.currentTimeMillis();
verified = verifySig(data2, keyPair.getPublic(), digitalSignature);
end = System.currentTimeMillis();
System.out.println("Verify: " + (end - start));
System.out.println(verified);
}
public static byte[] signData(byte[] data, PrivateKey key) throws Exception {
Signature signer = Signature.getInstance("SHA512withRSA");
signer.initSign(key);
signer.update(data);
return (signer.sign());
}
public static boolean verifySig(byte[] data, PublicKey key, byte[] sig) throws Exception {
Signature signer = Signature.getInstance("SHA512withRSA");
signer.initVerify(key);
signer.update(data);
return (signer.verify(sig));
}
public static KeyPair generateKeyPair(long seed) throws Exception {
KeyPairGenerator keyGenerator = KeyPairGenerator.getInstance("RSA");
SecureRandom rng = SecureRandom.getInstance("SHA1PRNG", "SUN");
rng.setSeed(seed);
keyGenerator.initialize(1024, rng);
return (keyGenerator.generateKeyPair());
}
}
This results in the output (time in ms)
KeyGen: 81
Sign: 52
Second Sign: 2
Verify: 2
I put some benchmarks on each line of the sign method, and it appears the overhead is purely from the sign action itself.
Create Sig Obj: 1
Init sign: 1
update: 0
Sign: 49
Create Sig Obj: 1
Init sign: 0
update: 0
Second Sign: 2
Any insight would be much appreciated
We're using nhibernate in and asp.net MVC application.
We are implementing the Session per Request pattern, via a httpModule.
It looks pretty straight forward, but when we run with NHibernate Profiler, it clearly shows that the
sessions are never getting closed.
the pattern seems straight forward...but I don't understand why the sessions are never closing.
here's the code i think is important.
set up the event handler:
context.EndRequest += new EventHandler(this.context_EndRequest);
in the handler dispose the Session
private void context_EndRequest(object sender, EventArgs e)
{
netLogHdl.ArchDebug("NHibernateHttpModule.context_EndRequest() ");
Dispose(0);// we are hitting 2 dbs and thus keep one session for each.
Dispose(1);
HttpContextBuildPolicy.DisposeAndClearAll();
}
private void Dispose(int sessionIndex)
{
netLogHdl.ArchStart("NHibernateHttpModule.Dispose", "int sessionIndex=\" + sessionIndex + \")");
try
{
//close the DB session
string sessManagerName = "";
string jcdcManager = "JCDC Manager";
string spamisManager = "Spamis Manager";
if (sessionIndex == 0)
sessManagerName = jcdcManager;
else
{
sessManagerName = spamisManager;
}
ISession oneSession = sessionPerDB[sessionIndex];
if (oneSession != null)
{
if (sessManagerName == jcdcManager) netLogHdl.ArchDebug(sessManagerName + " oneSession is NOT null");
if (oneSession.IsOpen)
{
// Don't flush - all saves should use transactions and calling Commit does the flush.
if (sessManagerName == jcdcManager) netLogHdl.ArchDebug(sessManagerName + " Closing the session");
//This will overrite it with the exact same session, if they don't match something weird is going on - EWB
oneSession = CurrentSessionContext.Unbind(factoryPerDB[sessionIndex]);
oneSession.Close();
}
else
{
if (sessManagerName == jcdcManager) netLogHdl.ArchDebug(sessManagerName + " Session is NOT open");
}
//if ( sessManagerName == jcdcManager ) netLogHdl.ArchDebug( sessManagerName + " Session got Dispose()-ing" );
//oneSession.Dispose();
}
else
{
if (sessManagerName == jcdcManager) netLogHdl.ArchDebug(sessManagerName + " Session is NULL");
}
sessionPerDB[sessionIndex] = null;
}
catch (Exception)
{
throw;
}
netLogHdl.ArchEnd();
}
Can anyone point me in the right direction? What shoud I look at, is the pattern not implemented correclty?
I'm flummoxed
Thanks!
E-
You should call dispose and not disconnect or close. System.IDisposable implementing objects should always be disposed by calling the dispose method, or by a using block.
You also might want to have a look at the code on Ben Day's blog post about session management.