I'm playing around with Asynchronous Apex using the Queueable interface. Salesforce does not allow more than 50 queued jobs running at one time. I want to do more than that so I decided to try and wait and queue more jobs as running jobs finish.
I have a helper class that I use to check if I can queue another job. Here is that code (I check for 40 so I have a buffer under the 50 limit).
public class QueueableHelper {
public static Boolean CanQueueJob()
{
List<AsyncApexJob> runningJobs = [SELECT Id FROM AsyncApexJob where JobType = 'Queueable' and Status = 'Processing'];
System.debug('Running jobs - ' + runningJobs.size());
return runningJobs.size() < 40;
}
}
If I run a loop to start 45 jobs and then later (while jobs are still running) call my helper method it will return false as expected. However if I try and use the helper method in a loop I hit the queue limit because the SOQL query in the helper method always returns 0. It's like the query isn't returning the latest data.
Here is the code I'm trying to use to make sure I never queue too many jobs.
for (Integer i = 0; i < 100; i++) {
while(!QueueableHelper.CanQueueJob())
{
System.debug('Waiting to queue - ' + i);
}
System.debug('Queueing job - ' + i);
System.enqueueJob(new QueueableExample());
}
Why is QueueableHelper.CanQueueJob never returning false?
Related
I am trying to consume a max of 1000 messages from kafka at a time. (I am doing this because i need to batch insert into MSSQL.) I was under the impression that kafka keeps an internal queue which fetches messages from the brokers and when i use the consumer.consume() method it just checks if there are any messages in the internal queue and returns if it finds something. otherwise it just blocks until the internal queue is updated or until timeout.
I tried to use the solution suggested here: https://github.com/confluentinc/confluent-kafka-dotnet/issues/1164#issuecomment-610308425
but when i specify TimeSpan.Zero (or any other timespan up to 1000ms) the consumer never consumes any messages. but if i remove the timeout it does consume messages but then i am unable to exit the loop if there are no more messages left to be read.
I also saw an other question on stackoverflow which suggested to read the offset of the last message sent to kafka and then read messages until i reach that offset and then break from the loop. but currently i only have one consumer and 6 partitions for a topic. I haven't tried it yet but i think managing offsets for each of the partition might make the code messy.
Can someone please tell me what to do?
static List<RealTime> getBatch()
{
var config = new ConsumerConfig
{
BootstrapServers = ConfigurationManager.AppSettings["BootstrapServers"],
GroupId = ConfigurationManager.AppSettings["ConsumerGroupID"],
AutoOffsetReset = AutoOffsetReset.Earliest,
};
List<RealTime> results = new List<RealTime>();
List<string> malformedJson = new List<string>();
using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
consumer.Subscribe("RealTimeTopic");
int count = 0;
while (count < batchSize)
{
var consumerResult = consumer.Consume(1000);
if (consumerResult?.Message is null)
{
break;
}
Console.WriteLine("read");
try
{
RealTime item = JsonSerializer.Deserialize<RealTime>(consumerResult.Message.Value);
results.Add(item);
count += 1;
}
catch(Exception e)
{
Console.WriteLine("malformed");
malformedJson.Add(consumerResult.Message.Value);
}
}
consumer.Close();
};
Console.WriteLine(malformedJson.Count);
return results;
}
I found a workaround.
For some reason the consumer first needs to be called without a timeout. That means it will wait for a message until it gets at least one. after that using consume with timeout zero fetches all the rest of the messages one by one from the internal queue. this seems to work out for the best.
I had a similar problem, updating the Confluent.Kafka and lidrdkafka libraries from version 1.8.2 to 2.0.2 helped
Scenario: Deactivate the user whose login date is less than 42 from today. I have an user whose last login date is 1/22/2020(US Date format)/22/1/2020 5:12 pm. Here I wrote a batch apex for deactivating. My code has executed successfully and my batch status is completed but the user record is not deactivating.
Here is the code:
global class User_Deactivation implements Database.Batchable<SObject>
{
dateTime dt = date.today()-42;
public String query = 'SELECT Name, LastLoginDate, Id From User WHERE IsActive = true AND LastLoginDate=:dt ';
global Database.querylocator start(Database.BatchableContext bc)
{
return Database.getQueryLocator(query);
}
global void execute(Database.BatchableContext bc,List<User> scope)
{
List<User> userList = new List<User>();
for(User s:scope)
{
User u =(user)s;
userList.add(u);
}
if(userList.size() > 0)
{
for(User usr : userList)
{
usr.isActive = false;
}
}
update userList;
}
global void finish(Database.BatchableContext bc)
{
AsyncApexJob a = [SELECT Id, Status, NumberOfErrors, JobItemsProcessed, TotalJobItems, CreatedBy.Email
FROM AsyncApexJob
WHERE Id = :BC.getJobId()];
Messaging.SingleEmailMessage mail = new Messaging.SingleEmailMessage();
String[] toAddresses = new String[] {a.CreatedBy.Email};
mail.setToAddresses(toAddresses);
mail.setSubject('Apex Job Status: ' + a.Status);
mail.setPlainTextBody('The batch Apex job processed ' + a.TotalJobItems + ' batches with '+ a.NumberOfErrors + ' failures.');
Messaging.sendEmail(new Messaging.SingleEmailMessage[] { mail });
}
}
please help me out on this
Multiple things you can improve here, where do I begin...
Initialisation(?) piece
dateTime dt = date.today()-42;
String query = 'SELECT Name, LastLoginDate, Id From User WHERE IsActive = true AND LastLoginDate=:dt';
Do you need Date or DateTime match? The way you wrote it it'll match only people who logged in exactly at midnight. System.debug(dt); would say 2020-01-23T00:00:00.000Z. It shouldn't be an equals sign, should be "less than" or "less or equal".
Or even better - you can make it bit more clear what you want to do, bit more "semantic" so the poor guy who's going to maintain it can understand it without extra comments. This reads more natural and uses the SOQL date literals, special "constants" to simplify your logic: SELECT Id, LastLoginDate FROM User WHERE isActive = true AND LastLoginDate != LAST_N_DAYS:42
What is this section of code anyway. It's not really static variables, it's not a constructor... I think it'll behave as a constructor. Be very, very careful with constructors for batches. The state of the class at the end of the constructor gets saved (serialised) and restored every time the class is scheduled to run. It's tempting to put some initialisation code into constructor, maybe read some custom settings, precalculate stuff... But then you'll be in for nasty surprise when admin adds new custom setting and the batch doesn't pick it up. In your case it's even worse, I'd suspect it'll serialise the dt and your today() will be frozen in time, not what you expected. To be safe move all initialisation logic to start()
And I'd even say whoever gave you the requirement didn't think it through. When you make new user they get a link they need to click in next 72h. If they didn't do it (maybe it was sent late Friday and they want to login on Monday) - this thing will dutifully kill their access at Friday night without giving them any chance to login. You need some "grace period". Maybe something like WHERE isActive = true AND (LastLoginDate < :x OR (LastLoginDate = null AND CreatedDate < :x))
start()
Queries in strings work and that's how a lot of batch documentation is written but they are poor practice. Where possible use a compiled query, in brackets. You get minimal improvement in execution (precompiled), you get compile-time warnings when you mess up (better than a runtime error which you might not notice if you don't monitor jobs). And most importantly - if somebody wants to delete a field - SF will detect a dependency and stop him/her. Use return Database.getQueryLocator([SELECT ...]); wherever you can.
execute()
Your scope already is a list of users, why do you do extra casts to User? Why do you add them to a helper list? Why 2 loops?
for(User u : scope){
u.isActive = false;
}
update users;
and you're done?
P.S. Why "global" all over the place?
Using Async-based Webservice and Async framework in WinRT (Win8) to get a large recordsets(1000 to 5000) from a remote Ms SQL Server.
I want to know :
1) Which is the best approach to handle to insert large recordsets into SQLite?
2) Using RollBack transaction will start all over again if there is connection error. The below method will insert whatever and I can update the data later if the records are not complete. Is this a good approach?
3) Any better way to enhance my below solution?
This foreach statement to handle
each reords in returned result which returned from Async-Based WebService:
foreach (WebServiceList _List in IList)
{
InsertNewItems(_List.No, _List.Description, _List.Unit_Price, _List.Base_Unit_of_Measure);
}
private void InsertNewItems(string ItemNo, string ItemName, decimal ItemPrice, string ItemBUoM)
{
var existingItem = (db2.Table().Where(c => c.No == ItemNo)).SingleOrDefault();
if (existingItem != null)
{
existingItem.No = ItemNo;
existingItem.Description = ItemName;
existingItem.Unit_Price = ItemPrice;
existingItem.BaseUnitofMeasure = ItemBUoM;
int success = db2.Update(existingItem);
}
else
{
int success = db2.Insert(new Item()
{
No = ItemNo,
Description = ItemName,
Unit_Price = ItemPrice,
Base_Unit_of_Measure = ItemBUoM
});
}
}
You should use RunInTransaction from sqlite-net. The documentation for it says,
Executes action within a (possibly nested) transaction by wrapping it
in a SAVEPOINT. If an exception occurs the whole transaction is rolled
back, not just the current savepoint. The exception is rethrown.
using (var db = new SQLiteConnection(DbPath))
{
db.RunInTransaction(() =>
{
db.InsertOrReplace(MyObj);
});
}
Wiki article for Transactions at GitHub
The most important performance aspect for bulk inserts is to use a single transaction. If you want to handle aborts, I suggest that you feed the data in sufficiently large parts and restart from that point on next time. An SQL transaction either finishes completely or rolls back completely, so unless the input data changes between two runs, there should be no need to do an insert-or-update.
See, for example, here for a discussion of SQLite bulk insert performance using different methods.
We are trying to load and resume workflows which have a delay. I have seen the Microsoft sample of Absolute Delay for this using store.WaitForEvents and LoadRunnableInstance to load the workflow. However here the workflow is already known.
In our case we want to have an event waiting for the store.WaitForEvents after every say 5 seconds to check if there is a runnable instance and if so only load and run that /those particular instances. Is there a way I could know which workflow instance is ready.
We are maintaing the workflow id and the xaml associated to it in our database, so if we could know the workflow instance id we could get the xaml mapped to it, create the workflow and then do a LOadRunnableInstance on it.
Any help would be greatly appreciated.
Microsoft sample (Absolute Delay)
public void Run(){
wfHostTypeName = XName.Get("Version" + Guid.NewGuid().ToString(),
typeof(WorkflowWithDelay).FullName);
this.instanceStore = SetupSqlpersistenceStore();
this.instanceHandle =
CreateInstanceStoreOwnerHandle(instanceStore, wfHostTypeName);
WorkflowApplication wfApp = CreateWorkflowApp();
wfApp.Run();
while (true)
{
this.waitHandler.WaitOne();
if (completed)
{
break;
}
WaitForRunnableInstance(this.instanceHandle);
wfApp = CreateWorkflowApp();
try
{
wfApp.LoadRunnableInstance();
waitHandler.Reset();
wfApp.Run();
}
catch (InstanceNotReadyException)
{
Console.WriteLine("Handled expected InstanceNotReadyException, retrying...");
}
}
Console.WriteLine("workflow completed.");
}
public void WaitForRunnableInstance(InstanceHandle handle)
{
var events=instanceStore.WaitForEvents(handle, TimeSpan.MaxValue);
bool foundRunnable = false;
foreach (var persistenceEvent in events)
{
if (persistenceEvent.Equals(HasRunnableWorkflowEvent.Value))
{
foundRunnable = true;
break;
}
}
if (!foundRunnable) {
Console.WriteLine("no runnable instance");
}
}
Thanks
Anamika
I had a similar problem with durable delay activities and WorkflowApplicationHost. Ended up creating my own 'Delay' activity that worked essentially the same way as the one out of the box, (takes an arg that describes when to resume the workflow, and then bookmarks itself). Instead of saving delay info in the SqlInstanceStore though, my Delay Activity created a record in a seperate db. (similar to the one you are using to track the Workflow Ids and Xaml). I then wrote a simple service that polled that DB for expired delays and initiated a resume of the necessary workflow.
Oh, and the Delay activity deleted it's record from that DB on bookmark resume.
HTH
I'd suggest having a separate SqlPersistenceStore for each workflow definition you're hosting.
This is for an enterprise web application.
We are building a front page/dashboard that queries our database for live statistics. The page has 3 update panels, each update panel has a user control that pulls data for it's box. Lets call the user controls UC_NotStarted, UC_Active and UC_Finished.
We have built the queries that return the data and all of them take a while to run (~7 seconds each). So if we let the first page run with one update panel the image spins for ~21 seconds and display everything. We have broken that code into 3 update panels and set the loading to conditional. When we do this we get one box to load every 7 seconds. That’s a step in the right direction but they load sequentially (UC_NotStarted is queried waits 7 seconds and displays, then UC_Active goes for 7 seconds, and finally UC_Finished runs for 7 seconds). We are still at 21 seconds but data is being shown every 7 seconds.
We would like all of the data to be shown in 7 seconds since all of the update panels should be fetching the data at the same time. Instead of using LinqToSQL to pull the data I am now leaning towards web services to get it but not sure if that will work.
Look at the ThreadPool, specifically the QueueUserWorkItem method. It's quite possible with this to run the 3 queries simultaneously and sit in a Thread.Sleep loop waiting for all three to finish executing so you can update all relevant parts of the page.
The following code will almost certainly require tweaking but would give you a rough idea of how to do this:
public class QueryResult
{
public bool Completed;
public DataTable Result;
public int ResultId;
}
public void GetData()
{
var QueryResults = new List<QueryResult>();
queryResults.Add(new QueryResult() { ResultId = 1 });
queryResults.Add(new QueryResult() { ResultId = 2 });
queryResults.Add(new QueryResult() { ResultId = 3 });
ThreadPool.QueueUserWorkItem(new WaitCallback(QueryRoutine), queryResults[0]);
ThreadPool.QueueUserWorkItem(new WaitCallback(QueryRoutine), queryResults[1]);
ThreadPool.QueueUserWorkItem(new WaitCallback(QueryRoutine), queryResults[2]);
var completedResults = new List<QueryResult>();
while(QueryResults.Count > 0)
{
for(int 1 = QueryResults.Count - 1; i >= 0; i--;)
{
if (queryResults[i].Completed)
{
completedResults.Add(queryResults[i]);
queryResults.RemoveAt(i);
}
}
Thread.Sleep(500);
}
// All background threads have completed, do something with the results,
// return them to the front-end, etc,..
}
public void QueryRoutine(object qR)
{
var queryResult = (QueryResult)qR;
// perform long running query here on a distinct thread, as QueryRoutine
// is now being run on three separate threads by the calls to QueueUserWorkItem
// based on queryResult.ResultId picking the relevant query for values
// of 1, 2 or 3
queryResult.Completed = true;
}