ASP.Net WebForms running mulitple queries at the same time - asp.net

This is for an enterprise web application.
We are building a front page/dashboard that queries our database for live statistics. The page has 3 update panels, each update panel has a user control that pulls data for it's box. Lets call the user controls UC_NotStarted, UC_Active and UC_Finished.
We have built the queries that return the data and all of them take a while to run (~7 seconds each). So if we let the first page run with one update panel the image spins for ~21 seconds and display everything. We have broken that code into 3 update panels and set the loading to conditional. When we do this we get one box to load every 7 seconds. That’s a step in the right direction but they load sequentially (UC_NotStarted is queried waits 7 seconds and displays, then UC_Active goes for 7 seconds, and finally UC_Finished runs for 7 seconds). We are still at 21 seconds but data is being shown every 7 seconds.
We would like all of the data to be shown in 7 seconds since all of the update panels should be fetching the data at the same time. Instead of using LinqToSQL to pull the data I am now leaning towards web services to get it but not sure if that will work.

Look at the ThreadPool, specifically the QueueUserWorkItem method. It's quite possible with this to run the 3 queries simultaneously and sit in a Thread.Sleep loop waiting for all three to finish executing so you can update all relevant parts of the page.
The following code will almost certainly require tweaking but would give you a rough idea of how to do this:
public class QueryResult
{
public bool Completed;
public DataTable Result;
public int ResultId;
}
public void GetData()
{
var QueryResults = new List<QueryResult>();
queryResults.Add(new QueryResult() { ResultId = 1 });
queryResults.Add(new QueryResult() { ResultId = 2 });
queryResults.Add(new QueryResult() { ResultId = 3 });
ThreadPool.QueueUserWorkItem(new WaitCallback(QueryRoutine), queryResults[0]);
ThreadPool.QueueUserWorkItem(new WaitCallback(QueryRoutine), queryResults[1]);
ThreadPool.QueueUserWorkItem(new WaitCallback(QueryRoutine), queryResults[2]);
var completedResults = new List<QueryResult>();
while(QueryResults.Count > 0)
{
for(int 1 = QueryResults.Count - 1; i >= 0; i--;)
{
if (queryResults[i].Completed)
{
completedResults.Add(queryResults[i]);
queryResults.RemoveAt(i);
}
}
Thread.Sleep(500);
}
// All background threads have completed, do something with the results,
// return them to the front-end, etc,..
}
public void QueryRoutine(object qR)
{
var queryResult = (QueryResult)qR;
// perform long running query here on a distinct thread, as QueryRoutine
// is now being run on three separate threads by the calls to QueueUserWorkItem
// based on queryResult.ResultId picking the relevant query for values
// of 1, 2 or 3
queryResult.Completed = true;
}

Related

SOQL query for running jobs not accurate

I'm playing around with Asynchronous Apex using the Queueable interface. Salesforce does not allow more than 50 queued jobs running at one time. I want to do more than that so I decided to try and wait and queue more jobs as running jobs finish.
I have a helper class that I use to check if I can queue another job. Here is that code (I check for 40 so I have a buffer under the 50 limit).
public class QueueableHelper {
public static Boolean CanQueueJob()
{
List<AsyncApexJob> runningJobs = [SELECT Id FROM AsyncApexJob where JobType = 'Queueable' and Status = 'Processing'];
System.debug('Running jobs - ' + runningJobs.size());
return runningJobs.size() < 40;
}
}
If I run a loop to start 45 jobs and then later (while jobs are still running) call my helper method it will return false as expected. However if I try and use the helper method in a loop I hit the queue limit because the SOQL query in the helper method always returns 0. It's like the query isn't returning the latest data.
Here is the code I'm trying to use to make sure I never queue too many jobs.
for (Integer i = 0; i < 100; i++) {
while(!QueueableHelper.CanQueueJob())
{
System.debug('Waiting to queue - ' + i);
}
System.debug('Queueing job - ' + i);
System.enqueueJob(new QueueableExample());
}
Why is QueueableHelper.CanQueueJob never returning false?

Loading any persistent workflow containing delay activity when it is a runnable instance in the store

We are trying to load and resume workflows which have a delay. I have seen the Microsoft sample of Absolute Delay for this using store.WaitForEvents and LoadRunnableInstance to load the workflow. However here the workflow is already known.
In our case we want to have an event waiting for the store.WaitForEvents after every say 5 seconds to check if there is a runnable instance and if so only load and run that /those particular instances. Is there a way I could know which workflow instance is ready.
We are maintaing the workflow id and the xaml associated to it in our database, so if we could know the workflow instance id we could get the xaml mapped to it, create the workflow and then do a LOadRunnableInstance on it.
Any help would be greatly appreciated.
Microsoft sample (Absolute Delay)
public void Run(){
wfHostTypeName = XName.Get("Version" + Guid.NewGuid().ToString(),
typeof(WorkflowWithDelay).FullName);
this.instanceStore = SetupSqlpersistenceStore();
this.instanceHandle =
CreateInstanceStoreOwnerHandle(instanceStore, wfHostTypeName);
WorkflowApplication wfApp = CreateWorkflowApp();
wfApp.Run();
while (true)
{
this.waitHandler.WaitOne();
if (completed)
{
break;
}
WaitForRunnableInstance(this.instanceHandle);
wfApp = CreateWorkflowApp();
try
{
wfApp.LoadRunnableInstance();
waitHandler.Reset();
wfApp.Run();
}
catch (InstanceNotReadyException)
{
Console.WriteLine("Handled expected InstanceNotReadyException, retrying...");
}
}
Console.WriteLine("workflow completed.");
}
public void WaitForRunnableInstance(InstanceHandle handle)
{
var events=instanceStore.WaitForEvents(handle, TimeSpan.MaxValue);
bool foundRunnable = false;
foreach (var persistenceEvent in events)
{
if (persistenceEvent.Equals(HasRunnableWorkflowEvent.Value))
{
foundRunnable = true;
break;
}
}
if (!foundRunnable) {
Console.WriteLine("no runnable instance");
}
}
Thanks
Anamika
I had a similar problem with durable delay activities and WorkflowApplicationHost. Ended up creating my own 'Delay' activity that worked essentially the same way as the one out of the box, (takes an arg that describes when to resume the workflow, and then bookmarks itself). Instead of saving delay info in the SqlInstanceStore though, my Delay Activity created a record in a seperate db. (similar to the one you are using to track the Workflow Ids and Xaml). I then wrote a simple service that polled that DB for expired delays and initiated a resume of the necessary workflow.
Oh, and the Delay activity deleted it's record from that DB on bookmark resume.
HTH
I'd suggest having a separate SqlPersistenceStore for each workflow definition you're hosting.

Is there any way to speed up crystal reports generation?

We are running a reporting web application that allows the user to select a few fields and a crystal report is generated based off of the fields selected. The SQL that is generated for the most complex report will return the data in < 5 seconds, however it takes the report and average of 3 minutes to run, sometimes longer causing a time out. We are running VS2010. The reports are basically set up out of the box with no real manipulations or computations being done, just displaying the data in a nice format. Is there anything we can try to speed it up, pre-loading a dummy report to load the dlls, some hack to make crystal run faster, anything?
EDIT: Code Added to show the databinding
protected void Page_Load(object sender, EventArgs e)
{
if (!Page.IsPostBack)
{
string strFile = Server.MapPath(#"AwardStatus.rpt");
CrystalReportSource1.Report.FileName = strFile;
DataTable main = Main();
CrystalReportSource1.ReportDocument.SetDataSource(main);
CrystalReportViewer1.HasCrystalLogo = false;
CrystalReportSource1.ReportDocument.ExportToHttpResponse(CrystalDecisions.Shared.ExportFormatType.PortableDocFormat, Response, false, "pmperformance");
}
}
private DataTable Main()
{
Guid guidOffice = Office;
CMS.Model.ReportsTableAdapters.ViewACTableAdapter rptAdapter = new CMS.Model.ReportsTableAdapters.ViewACTableAdapter();
Reports.ViewAwardedContractsDataTable main = new Reports.ViewAwardedContractsDataTable();
if (Office == new Guid())
{
IEnumerable<DataRow> data = rptAdapter.GetData().Where(d => UserPermissions.HasAccessToOrg(d.guidFromId, AuthenticatedUser.PersonID)).Select(d => d);
foreach (var row in data)
{
main.ImportRow(row);
}
}
else if (guidOffice != new Guid())
{
main = rptAdapter.GetDataByOffice(guidOffice);
}
else
{
main = new Reports.ViewACDataTable();
}
return main;
}
private Guid Office
{
get
{
string strOffice = Request.QueryString["Office"];
Guid guidOffice = BaseControl.ParseGuid(strOffice);
if (!UserPermissions.HasAccessToOrg(guidOffice, AuthenticatedUser.PersonID))
{
return Guid.Empty;
}
else
{
return guidOffice;
}
}
}
protected void CrystalReportSource1_DataBinding(object sender, EventArgs e)
{
//TODO
}
This may be a bit flippant, but possibly consider not using crystal reports... We had a fair bit of trouble with them recently (out of memory errors being one), and we've moved off to other options and are quite happy...
Here's what I would do:
Put clocks from the time you get the field choices from the user, all the way to when you display the report. See where your processing time is going up.
When you look at the clocks, there can be various situations:
If Crystal Reports is taking time to fill the report, check how you're filling it. If you're linking the report fields directly to your data table, CR is probably taking time looking up the data. I suggest creating a new table (t_rpt) with dynamic columns (Field1, Field2,..FieldN) and pointing your report template to that table. I don't know if you're already doing this.
If it's taking time for you to lookup the data itself, I suggest creating a view of your table. Even though a memory hog, this will make your lookup quick and you can delete the view once you're done.
If it's none of the above, let us know what your clocks show.
In terms of loading any large amount of data, you'll always want to use a stored procedure.
Outside of that, you WILL see a delay in the report running the first time the Crystal DLLs load. Yes, you can preload them as you mentioned and that will help some.

Random failures using CMISQL queries on Alfresco 3.3.0

[Solved, it seems that there was some bug affecting Alfresco 3.3.0, which is no longer present on Alfresco 3.3.0g]
Hi,
I'm using OpenCMIS to retrieve data from Alfresco 3.3, but it's having a very weird behaviour on CMISQL queries. I've googled somebody else with the same problems, but it seems I'm the first one all over the world :), so I guess it's my fault, not OpenCMIS'.
This is how I'm querying Alfresco:
public Class CmisTest {
private static Session sesion;
private static final String QUERY = "select cmis:objectid, cmis:name from cmis:folder where cmis:name='MyFolder'";
public static void main(String[] args) {
// Open a CMIS session with Alfresco
Map<String, String> params = new HashMap<String, String>();
params.put(SessionParameter.USER, "admin");
params.put(SessionParameter.PASSWORD, "admin");
params.put(SessionParameter.ATOMPUB_URL, "http://localhost:8080/alfresco/s/api/cmis");
params.put(SessionParameter.BINDING_TYPE, BindingType.ATOMPUB.value());
params.put(SessionParameter.REPOSITORY_ID, "fa9d2553-1e4d-491b-87fd-3de894dc7ca9");
sesion = SessionFactoryImpl.newInstance().createSession(params);
// Ugly bug in Alfresco which raises an exception if we request more data than it's available
// See https://issues.alfresco.com/jira/browse/ALF-2859
sesion.getDefaultContext().setMaxItemsPerPage(1);
// We repeat the same query 20 times and count the number of elements retrieved each time
for (int i = 0; i < 20; i++) {
List<QueryResult> result = doQuery();
System.out.println(result.size() + " folders retrieved");
}
}
public static List<QueryResult> doQuery() {
List<QueryResult> result = new LinkedList<QueryResult>();
try {
int page = 0;
while (true) {
ItemIterable<QueryResult> iterable = sesion.query(QUERY, false).skipTo(page);
page++;
for (QueryResult qr : iterable) {
result.add(qr);
}
}
} catch (Exception e) {
// We will always get an exception when Alfresco has no more data to retrieve... :(
// See https://issues.alfresco.com/jira/browse/ALF-2859
}
return result;
}
}
As you can see, we just execute the same query, up to 20 times in a row. You would expect the same result each time, wouldn't you? Unfortunately, this is a sample of what we get:
1 folders retrieved
1 folders retrieved
1 folders retrieved
0 folders retrieved
0 folders retrieved
0 folders retrieved
0 folders retrieved
0 folders retrieved
1 folders retrieved
1 folders retrieved
Sometimes we get 20 1 in a row, sometimes it's all 0. We have never get a "mix" of 1 and 0, though; we always get "a run" of them.
It does not matter if we create the session before each query, we still have the random issue. We have tried against two different Alfresco servers (both of them 3.3 Community), clean installation, and they both fail randomly. We also tried to measure the time for each query, but it doesn't seem to have any relation with the result being wrong (0 folders retrieved) or right (1 folders retrieved).
Alfresco seems to be working fine: if we go to "Administration --> Node browser" and launch the CMISQL query from there, it always retrieves one folder, which is right. So, it must be our code, or an OpenCMIS bug...
Any ideas?
I can't reproduce this behavior. It's running fine against http://cmis.alfresco.com . The issue https://issues.alfresco.com/jira/browse/ALF-2859 states that there have been bug fixes. Are you running the latest Alfresco version?
Florian

ASP.NET: HttpModule performance

I've implemented an HttpModule that intercepts the Response stream of every request and runs a half dozen to a dozen Regex.Replace()s on each text/html-typed response. I'm concerned about how much of a performance hit I'm incurring here. What's a good way to find out? I want to compare speed with and without this HttpModule running.
I've a few of these that hook into the Response.Filter stream pipeline to provide resource file integration, JS/CSS packing and rewriting of static files to absolute paths.
As long as you test your regexes in RegexBuddy for speed over a few million iterations, ensure you use RegexOptions.Compiled, and remember that often the quickest and most efficient technique is to use a regex to broadly identify matches and then use C# to hone that to exactly what you need.
Make sure you're also caching and configuration that you rely upon.
We've had a lot of success with this.
Http module is just common piece of code, so you can measure time of execution of this particular regex replace stuff. It is enough. Have a set of typical response streams as input of your stress test and measure executing of the replace using Stopwatch class. Consider also RegexOptions.Compiled switch.
Here are a few ideas:
Add some Windows performance counters, and use them to measure and report average timing data. You might also increment a counter only if the time measurement exceeds a certain threshold. and
Use tracing combined with Failed Request Tracing to collect and report timing data. You can also trigger FRT reports only if page execution time exceeds a threshold.
Write a unit test that uses the Windows OS clock to measure how long your code takes to execute.
Add a flag to your code that you can turn on or off with a test page to enable or disable your regex code, to allow easy A/B testing.
Use a load test tool like WCAT to see how many page requests per second you can process with and without the code enabled.
I recently had to do some pef tests on an HTTPModule that I wrote and decided to perform a couple of load tests to simulate web traffic and capture the performance times with and without the module configured. It was the only way I could figure to really know the affect of having the module installed.
I would usually do something with Apache Bench (see the following for how to intsall, How to install apache bench on windows 7?), but I had to also use windows authentication. As ab only has basic authentication I it wasn't a fit for me. ab is slick and allows for different request scenarios, so that would be the first place to look. One other thought is you can get a lot of visibility by using glimpse as well.
Being that I couldn't use ab I wrote something custom that will allow for concurrent requests and test different url times.
Below is what I came up with to test the module, hope it helps!
// https://www.nuget.org/packages/RestSharp
using RestSharp;
using RestSharp.Authenticators;
using RestSharp.Authenticators.OAuth;
using RestSharp.Contrib;
using RestSharp.Deserializers;
using RestSharp.Extensions;
using RestSharp.Serializers;
using RestSharp.Validation;
string baseUrl = "http://localhost/";
void Main()
{
for(var i = 0; i < 10; i++)
{
RunTests();
}
}
private void RunTests()
{
var sites = new string[] {
"/resource/location",
};
RunFor(sites);
}
private void RunFor(string[] sites)
{
RunTest(sites, 1);
RunTest(sites, 5);
RunTest(sites, 25);
RunTest(sites, 50);
RunTest(sites, 100);
RunTest(sites, 500);
RunTest(sites, 1000);
}
private void RunTest(string[] sites, int iterations, string description = "")
{
var action = GetAction();
var watch = new Stopwatch();
// Construct started tasks
Task<bool>[] tasks = new Task<bool>[sites.Count()];
watch.Start();
for(int j = 0; j < iterations; j++)
{
for (int i = 0; i < sites.Count(); i++)
{
tasks[i] = Task<bool>.Factory.StartNew(action, sites[i]);
}
}
try
{
Task.WaitAll(tasks);
}
catch (AggregateException e)
{
Console.WriteLine("\nThe following exceptions have been thrown by WaitAll()");
for (int j = 0; j < e.InnerExceptions.Count; j++)
{
Console.WriteLine("\n-------------------------------------------------\n{0}", e.InnerExceptions[j].ToString());
}
}
finally
{
watch.Stop();
Console.WriteLine("\"{0}|{1}|{2}\", ",sites.Count(), iterations, watch.Elapsed.TotalSeconds);
}
}
private Func<object, bool> GetAction()
{
baseUrl = baseUrl.Trim('/');
return (object obj) =>
{
var str = (string)obj;
var client = new RestClient(baseUrl);
client.Authenticator = new NtlmAuthenticator();
var request = new RestRequest(str, Method.GET);
request.AddHeader("Accept", "text/html");
var response = client.Execute(request);
return (response != null);
};
}

Resources