Memory limit for running external executables within Asp.net - asp.net

I am using WkhtmltoPdf in my C# web application running in .NET 4.0 to generate PDFs from HTML files. In general everything works fine except when the size of the HTML file is below 250KB. Once the HTML file size increases beyond that, the process which runs the wkhtmltopdf.exe gives an exception as below. On the Task Manager, I have seen that the Memory value for the wkhtmltopdf.exe process does not increase beyond a value of 40,096 K, which I believe is the reason why the process is abandoned in between.
How can we configure such that the memory limit for external exes can be increased? Is there any other way of solving this issue?
More info:
When I run the conversion from the command line directly, the PDF is generated fine. So, its unlikely to be a problem with WkhtmlToPdf.
The error is from localhost. I have tried the same on the DEV server, with the same result.
EDIT:
More specific exception message: - For the MainModule property of the
Process object, the error says - {"Only part of a ReadProcessMemory or
WriteProcessMemory request was completed"}, with the NativeErrorCode
value - 299.
Exception:
> [Exception: Loading pages (1/6) [>
> ] 0% [======> ]
> 10% [======> ] 11%
> [=======> ] 13%
> [=========> ] 15%
> [==========> ] 18%
> [============> ] 20%
> [=============> ] 22%
> [==============> ] 24%
> [===============> ] 26%
> [=================> ] 29%
> [==================> ] 31%
> [===================> ] 33%
> [=====================> ] 35%
> [======================> ] 37%
> [========================> ] 40%
> [=========================> ] 42%
> [==========================> ] 44%
> [============================> ] 47%
> [=============================> ] 49%
> [==============================> ] 51%
> [============================================================] 100%
> Counting pages (2/6)
> [============================================================] Object
> 1 of 1 Resolving links (4/6)
> [============================================================] Object
> 1 of 1 Loading headers and footers (5/6)
> Printing pages (6/6) [>
> ] Preparing [=>
> ] Page 1 of 49 [==>
> ] Page 2 of 49 [===>
> ] Page 3 of 49 [====>
> ] Page 4 of 49 [======>
> ] Page 5 of 49 [=======>
> ] Page 6 of 49 [========>
> ] Page 7 of 49 [=========>
> ] Page 8 of 49 [==========>
> ] Page 9 of 49 [============>
> ] Page 10 of 49 [=============>
> ] Page 11 of 49 [==============>
> ] Page 12 of 49 [===============>
> ] Page 13 of 49 [================>
> ] Page 14 of 49 [==================>
> ] Page 15 of 49 [===================>
> ] Page 16 of 49 [====================>
> ] Page 17 of 49 [=====================>
> ] Page 18 of 49 [======================>
> ] Page 19 of 49 [========================>
> ] Page 20 of 49 [=========================>
> ] Page 21 of 49 [==========================>
> ] Page 22 of 49 [===========================>
> ] Page 23 of 49 [============================>
> ] Page 24 of 49 [==============================>
> ] Page 25 of 49 [===============================>
> ] Page 26 of 49 [=================================>
> ] Page 27 of 49 [==================================>
> ]
Code that I use:
var fileName = " - ";
var wkhtmlDir = ConfigurationManager.AppSettings[Constants.AppSettings.ExportToPdfExecutablePath];
var wkhtml = ConfigurationManager.AppSettings[Constants.AppSettings.ExportToPdfExecutablePath] + "\\wkhtmltopdf.exe";
var p = new Process();
string switches = "";
switches += "--print-media-type ";
switches += "--margin-top 10mm --margin-bottom 10mm --margin-right 5mm --margin-left 5mm ";
switches += "--page-size A4 ";
switches += "--disable-smart-shrinking ";
var startInfo = new ProcessStartInfo
{
CreateNoWindow = true,
FileName = wkhtml,
Arguments = switches + " " + url + " " + fileName,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput=true,
WorkingDirectory=wkhtmlDir
};
p.StartInfo = startInfo;
p.Start();
Debugger Screenshot of WkHtmlToPdf.exe Process:

This is what your looking for:
http://jobobjectwrapper.codeplex.com/
I couldn't find anything else that pertains to "increasing" the Memory Limit for a process, although I've heard of people limiting the Process Memory with MaxWorkingSet, but I believe this is only for virtual memory after the application has used all it can.
The Job Object is a good place to start, they're just a collection of processes that are easily controllable.
"With this library you can create job objects, create and assign a
process to the job, control process and job limits, and register for
the various process- and job-related notification events."
This might be of use too:
Calling wkhtmltopdf to generate PDF from HTML

Have a look at this to see whether the Threads Per Process limit may be coming in to play here. It's a long shot (and I'm not aware of any memory limit imposed by IIS on external processes), but note this from the documentation:
Because this property defines the maximum number of ASP requests that
can execute simultaneously, this setting should remain at the default
value unless your ASP applications are making extended calls to
external components. In this case, you may increase the value of
Threads Per Processor Limit. Doing so allows the server to create more
threads to handle more concurrent requests.

Related

Set multiple threshold on a log based kusto query

I have set up a log-based alert in Microsoft Azure. The deployment of the alerts done via ARM template.
Where you can input your query and set threshold like below.
"triggerThresholdOperator": {
"value": "GreaterThan"
},
"triggerThreshold": {
"value": 0
},
"frequencyInMinutes": {
"value":15
},
"timeWindowInMinutes": {
"value": 15
},
"severityLevel": {
"value": "0"
},
"appInsightsQuery": {
"value": "exceptions\r\n| where A_ != '2000' \r\n| where A_ != '4000' \r\n| where A_ != '3000' "
}
As far as I understand we can only set threshold once ON an entire query.
Questions: I have multiple statements in my query which I am excluding since it's just a noise. But now I want to set a threshold on value 3000 to 5 and also want to set a time-window to 30 in the same query. meaning only exclude 3000 when it occurs 5 times in the last 30 minutes(when query get run).
exceptions
| where A_ != '2000'
| where A_ != '4000'
| where A_ != '3000'
I am pretty sure that I can't set a threshold like this in the query and the only workaround is to create a new alert just for value 3000 and set a threshold in ARM template. I haven't found any heavy threshold/time filters in Aure. Is there any way I can set multiple thresholds and time filters in a single query? which is again getting checked by different threshold and time filetrs in the ARM template.
Thanks.
I don't fully understand your question.
But for your time window question you could do something like
exceptions
| summarize count() by A_, bin(TimeGenerated, 30m)
That way you will get a count of A_ in blocks of 30 minutes.
Another way would be to do:
let Materialized = materialize(
exceptions
| summarize Count=count(A_) by bin(TimeGenerated, 30m)
); 
Materialized | where Count == 10
But then again it all depends on what you would like to achieve
You can easily set that in the query and fire based on the aggregate result.
exceptions
| where timestamp > ago(30m)
| summarize count2000 = countif(A_ == '2000'), count3000 = countif(A_ == '3000'), count4000 = countif(A_ == '4000')
| where count2000 > 5 or count3000 > 3 or count4000 > 4
If the number of results is greater than one than the aggregate condition applies.

How to resolve error " Hunk #2 FAILED at 456. 1 out of 2 hunks FAILED"

I am trying to run the following command in ubuntu terminal
patch -p0 -i adjustmentFile.patch
That is giving the following error
patching file ./src/helpStructures/CastaliaModule.cc
patching file ./src/node/communication/mac/tunableMac/TunableMAC.cc
Hunk #2 FAILED at 456.
1 out of 2 hunks FAILED -- saving rejects to file ./src/node/communication/mac/tunableMac/TunableMAC.cc.rej
I tried almost all the ways suggested in the link Hunk #1 FAILED at 1. What's that mean?. However, nothing worked.
Here is my version detail
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Jun 06 2019 17:31:41)
Included patches: 1-1453
The patch file:
diff -r -u ./src/helpStructures/CastaliaModule.cc ./src/helpStructures/CastaliaModule.cc
--- ./src/helpStructures/CastaliaModule.cc 2010-12-09 09:56:47.000000000 -0300
+++ ./src/helpStructures/CastaliaModule.cc 2011-12-20 00:16:39.944320051 -0300
## -180,6 +180,8 ##
classPointers.resourceManager = getParentModule()->getParentModule()->getSubmodule("ResourceManager");
else if (name.compare("SensorManager") == 0)
classPointers.resourceManager = getParentModule()->getSubmodule("ResourceManager");
+ else if (name.compare("Routing") == 0)
+ classPointers.resourceManager = getParentModule()->getParentModule()->getSubmodule("ResourceManager");
else
opp_error("%s module has no rights to call drawPower() function", getFullPath().c_str());
if (!classPointers.resourceManager)
Only in ./src/helpStructures: CastaliaModule.cc~
diff -r -u ./src/node/communication/mac/tunableMac/TunableMAC.cc ./src/node/communication/mac/tunableMac/TunableMAC.cc
--- ./src/node/communication/mac/tunableMac/TunableMAC.cc 2011-03-30 02:14:34.000000000 -0300
+++ ./src/node/communication/mac/tunableMac/TunableMAC.cc 2011-12-19 23:57:43.894686687 -0300
## -405,6 +405,8 ##
void TunableMAC::fromRadioLayer(cPacket * pkt, double rssi, double lqi)
{
TunableMacPacket *macFrame = dynamic_cast <TunableMacPacket*>(pkt);
+ macFrame->getMacRadioInfoExchange().RSSI = rssi;
+ macFrame->getMacRadioInfoExchange().LQI = lqi;
if (macFrame == NULL){
collectOutput("TunableMAC packet breakdown", "filtered, other MAC");
return;
## -454,7 +456,8 ##
}
case DATA_FRAME:{
- toNetworkLayer(macFrame->decapsulate());
+ cPacket *netPkt = decapsulatePacket(macFrame);
+ toNetworkLayer(netPkt);
collectOutput("TunableMAC packet breakdown", "received data pkts");
if (macState == MAC_STATE_RX) {
cancelTimer(ATTEMPT_TX);
Only in ./src/node/communication/mac/tunableMac: TunableMAC.cc~
Patching takes some changes made to a file X, and applies them to a different instance of file X. That is, suppose you start with generation 1 of file X; you make changes to get generation 2-a, and someone else starts with generation 1 to make generation 2-b. Now you want to take his edits that created his generation 2-b, and apply them to your generation 2-a.
If 'his' changes clash with 'your' changes, they cannot be automatically patched.
You'll need to look at the changes being made in hunk 2.
- toNetworkLayer(macFrame->decapsulate());
+ cPacket *netPkt = decapsulatePacket(macFrame);
+ toNetworkLayer(netPkt);
and figure out what you want the result to look like. Someone needs to know what the result is supposed to be. You can't resolve conflicts without knowledge of intent.

Two index with one value in a lua table

I am very new to lua and my plan is to create a table. This table (I call it test) has 200 entries - each entry has the same subentries (In this example the subentries money and age):
This is a sort of pseudocode:
table test = {
Entry 1: money=5 age=32
Entry 2: money=-5 age=14
...
Entry 200: money=999 age=72
}
How can I write this in lua ? Is there a possibility ? The other way would be, that I write each subentry as a single table:
table money = { }
table age = { }
But for me, this isn't a nice way, so maybe you can help me.
Edit:
This question Table inside a table is related, but I cannot write this 200x.
Try this syntax:
test = {
{ money = 5, age = 32 },
{ money = -5, age = 14 },
...
{ money = 999, age = 72 }
}
Examples of use:
-- money of the second entry:
print(test[2].money) -- prints "-5"
-- age of the last entry:
print(test[200].age) -- prints "72"
You can also turn the problem on it's side, and have 2 sequences in test: money and age where each entry has the same index in both arrays.
test = {
money ={1000,100,0,50},
age={40,30,20,25}
}
This will have better performance since you only have the overhead of 3 tables instead of n+1 tables, where n is the number of entries.
Anyway you have to enter your data one way or another. What you'd typically do is make use some easily parsed format like CSV, XML, ... and convert that to a table. Like this:
s=[[
1000 40
100 30
0 20
50 25]]
test ={ money={},age={}}
n=1
for balance,age in s:gmatch('([%d.]+)%s+([%d.]+)') do
test.money[n],test.age[n]=balance,age
n=n+1
end
You mean you do not want to write "money" and "age" 200x?
There are several solutions but you could write something like:
local test0 = {
5, 32,
-5, 14,
...
}
local test = {}
for i=1,#test0/2 do
test[i] = {money = test0[2*i-1], age = test0[2*i]}
end
Otherwise you could always use metatables and create a class that behaves exactly like you want.

Get component of Date / ISODate in mongo

How to get a component like minute from ISODate stored in MongoCollection?
Since you don't specify a language, I'm going to assume you mean JavaScript, as in the shell.
One of the nice features of the shell is it has tab completion. So you can do something like this:
> db.test.insert({x:new Date()});
> var doc = db.test.findOne();
> doc
{
"_id" : ObjectId("4fa131851932655dc45027a9"),
"x" : ISODate("2012-05-02T13:07:17.012Z")
}
> doc.x
ISODate("2012-05-02T13:07:17.012Z")
> doc.x.<TAB><TAB>
doc.x.constructor doc.x.getSeconds( doc.x.getUTCMinutes( doc.x.setHours( doc.x.setUTCHours( doc.x.toLocaleDateString(
doc.x.getDate( doc.x.getTime( doc.x.getUTCMonth( doc.x.setMilliseconds( doc.x.setUTCMilliseconds( doc.x.toLocaleString(
doc.x.getDay( doc.x.getTimezoneOffset( doc.x.getUTCSeconds( doc.x.setMinutes( doc.x.setUTCMinutes( doc.x.toLocaleTimeString(
doc.x.getFullYear( doc.x.getUTCDate( doc.x.getYear( doc.x.setMonth( doc.x.setUTCMonth( doc.x.toString(
doc.x.getHours( doc.x.getUTCDay( doc.x.hasOwnProperty( doc.x.setSeconds( doc.x.setUTCSeconds( doc.x.toTimeString(
doc.x.getMilliseconds( doc.x.getUTCFullYear( doc.x.propertyIsEnumerable( doc.x.setTime( doc.x.setYear( doc.x.toUTCString(
doc.x.getMinutes( doc.x.getUTCHours( doc.x.setDate( doc.x.setUTCDate( doc.x.toDateString( doc.x.tojson(
doc.x.getMonth( doc.x.getUTCMilliseconds( doc.x.setFullYear( doc.x.setUTCFullYear( doc.x.toGMTString( doc.x.valueOf(
What you want is probably:
> doc.x.getSeconds();
17
> doc.x.getMinutes();
7
> doc.x.getHours();
9
> doc.x.getDate();
2
> doc.x.getMonth();
4
> doc.x.getFullYear();
2012

Programming Logic - Splitting up Tasks Between Threads

Lets say you want 5 threads to process data simultaneous. Also assume, you have 89 tasks to process.
Off the bat you know 89 / 5 = 17 with a remainder of 4. The best way to split up tasks would be to have 4 (the remainder) threads process 18 (17+1) tasks each and then have 1 (# threads - remainder) thread to process 17.
This will eliminate the remainder. Just to verify:
Thread 1: Tasks 1-18 (18 tasks)
Thread 2: Tasks 19-36 (18 tasks)
Thread 3: Tasks 37-54 (18 tasks)
Thread 4: Tasks 55-72 (18 tasks)
Thread 5: Tasks 73-89 (17 tasks)
Giving you a total of 89 tasks completed.
I need a way of getting the start and ending range of each thread mathematically/programmability; where the following should print the exact thing I have listed above:
$NumTasks = 89
$NumThreads = 5
$Remainder = $NumTasks % $NumThreads
$DefaultNumTasksAssigned = floor($NumTasks / $NumThreads)
For $i = 1 To $NumThreads
if $i <= $Remainder Then
$NumTasksAssigned = $DefaultNumTasksAssigned + 1
else
$NumTasksAssigned = $DefaultNumTasksAssigned
endif
$Start = ??????????
$End = ??????????
print Thread $i: Tasks $Start-$End ($NumTasksAssigned tasks)
Next
This should also work for any number of $NumTasks.
Note: Please stick to answering the math at hand and avoid suggesting or assuming the situation.
Why? Rather then predetermining the scheduling order, stick all of the tasks on a queue, and then have each thread pull them off one by one when they're ready. Then your tasks will basically run "as fast as possible".
If you pre-allocated, then one thread may be doing a particularly long bit of processing and blocking the running of all the tasks stuck behind it. Using the queue, as each task finishes and a thread frees up, it grabs the next task and keeps going.
Think of it like a bank with 1 line per teller vs one line and a lot of tellers. In the former, you might get stuck behind the person depositing coins and counting it out one by one, the latter you get to the next available teller, while Mr. PocketChange counts away.
I second Will Hartung 's remark. You may just feed them one task at a time (or a few tasks at a at a time, depending if there's much overhead, i.e. if individual tasks get typically completed very fast, relative to the cost of starting/recycling threads). Your subsequent comments effectively explain that your "threads" carry a heavy creation cost, and hence your desire to feed them once with as much work as possible, rather than wasting time creating new "thread" each fed a small amount of work.
Anyway... going to the math question...
If you'd like to assign tasks just once, the following formula, plugged in lieu of the the ????????? in your logic, should do the trick:
$Start = 1
+ (($i -1) * ($DefaultNumTasksAssigned + 1)
- (floor($i / ($Remainder + 1)) * ($i - $Remainder))
$End = $Start + $NumTasksAssigned -1
The formula is explained as follow:
1 is for the fact that your display / logic is one-based not zero-based
The second term is because we generally add ($DefaultNumTasksAssigned + 1) with each iteration.
The third term provides a correction for the last few iterations.
Its first part, (floor($i / ($Remainder + 1)) provides 0 until $i reaches the first thread
that doesn't receive one extra task, and 1 thereafter.
The second part express by how much we need to correct.
The formula for $End is easier, the only trick is the minus 1, it is because the Start and End values are inclusive (so for example between 1 an 19 there are 19 tasks not 18)
The following slightly modified piece of logic should also work, it avoids the "fancy" formula by keeping a running tab of the $Start variable, rather than recomputing it each time..
$NumTasks = 89
$NumThreads = 5
$Remainder = $NumTasks % $NumThreads
$DefaultNumTasksAssigned = floor($NumTasks / $NumThreads)
$Start = 1
For $i = 1 To $NumThreads
if $i <= $Remainder Then // fixed here! need <= because $i is one-based
$NumTasksAssigned = $DefaultNumTasksAssigned + 1
else
$NumTasksAssigned = $DefaultNumTasksAssigned
endif
$End = $Start + $NumTasksAssigned -1
print Thread $i: Tasks $Start-$End ($NumTasksAssigned tasks)
$Start = $Start + $NumTasksAssigned
Next
Here's a Python transcription of the above
>>> def ShowWorkAllocation(NumTasks, NumThreads):
... Remainder = NumTasks % NumThreads
... DefaultNumTasksAssigned = math.floor(NumTasks / NumThreads)
... Start = 1
... for i in range(1, NumThreads + 1):
... if i <= Remainder:
... NumTasksAssigned = DefaultNumTasksAssigned + 1
... else:
... NumTasksAssigned = DefaultNumTasksAssigned
... End = Start + NumTasksAssigned - 1
... print("Thread ", i, ": Tasks ", Start, "-", End, "(", NumTasksAssigned,")")
... Start = Start + NumTasksAssigned
...
>>>
>>> ShowWorkAllocation(89, 5)
Thread 1 : Tasks 1 - 18 ( 18 )
Thread 2 : Tasks 19 - 36 ( 18 )
Thread 3 : Tasks 37 - 54 ( 18 )
Thread 4 : Tasks 55 - 72 ( 18 )
Thread 5 : Tasks 73 - 89 ( 17 )
>>> ShowWorkAllocation(11, 5)
Thread 1 : Tasks 1 - 3 ( 3 )
Thread 2 : Tasks 4 - 5 ( 2 )
Thread 3 : Tasks 6 - 7 ( 2 )
Thread 4 : Tasks 8 - 9 ( 2 )
Thread 5 : Tasks 10 - 11 ( 2 )
>>>
>>> ShowWorkAllocation(89, 11)
Thread 1 : Tasks 1 - 9 ( 9 )
Thread 2 : Tasks 10 - 17 ( 8 )
Thread 3 : Tasks 18 - 25 ( 8 )
Thread 4 : Tasks 26 - 33 ( 8 )
Thread 5 : Tasks 34 - 41 ( 8 )
Thread 6 : Tasks 42 - 49 ( 8 )
Thread 7 : Tasks 50 - 57 ( 8 )
Thread 8 : Tasks 58 - 65 ( 8 )
Thread 9 : Tasks 66 - 73 ( 8 )
Thread 10 : Tasks 74 - 81 ( 8 )
Thread 11 : Tasks 82 - 89 ( 8 )
>>>
I think you've solved the wrong half of your problem.
It's going to be virtually impossible to precisely determine the time it will take to complete all your tasks, unless all of the following are true:
your tasks are 100% CPU-bound: that is, they use 100% CPU while running and don't need to do any I/O
none of your tasks have to synchronize with any of your other tasks in any way
you have exactly as many threads as you have CPUs
the computer that is running these tasks is not performing any other interesting tasks at the same time
In practice, most of the time, your tasks are I/O-bound rather than CPU-bound: that is, you are waiting for some external resource such as reading from a file, fetching from a database, or communicating with a remote computer. In that case, you only make things worse by adding more threads, because they're all contending for the same scarce resource.
Finally, unless you have some really weird hardware, it's unlikely you can actually have exactly five threads running simultaneously. (Usually processor configurations come in multiples of at least two.) Usually the sweet spot is at about 1 thread per CPU if your tasks are very CPU-bound, about 2 threads per CPU if the tasks spend half their time being CPU-bound and half their time doing IO, etc.
tl;dr: We need to know a lot more about what your tasks and hardware look like before we can advise you on this question.

Resources