I am getting the CKR_DEVICE_MEMORY error code for C_Encrypt call using cryptoki library.
From the PKCS#11 spec, CKR_DEVICE_MEMORY means The token doesnot have sufficient memory to perform the requested function.
Under what circumstances, do we get the token's memory completely full?
The HSM has been working 24x7 for 7 days continuously mostly encrypting and decrypting files during the day time with 2 parallel sessions. I haven't called C_Finalize in the last 7 days. so cryptoki library has been working in its memory space from the point it has been initialised(see a related post on this).
I can see from my applications, debug log, what ever, i am allocating, i am deallocating so there is no memory leak from my application code.
UPDATE 1: There is a related detailed discussion on how i can call C_Finalize in Application_Endof the ASP.NET. The main reason i couldn't use this because after recycling/timeout, the ASP.net threads access a single session resulting in CKR_OPERATION_ACTIVE error. In my case multiple applications are accessing the HSM via a webservice.
Let’s consider the memory of HSM and host computer (which Cryptoki library runs on) separately. Correctly implemented Cryptoki library should return CKR_DEVICE_MEMORY if HSM device suffers from the lack of memory and CKR_HOST_MEMORY if Cryptoki library can’t allocate host computer memory for its internal structures (if it is implemented as a shared library then the process can’t allocate memory). So if your pkcs11 library is implemented correctly then CKR_DEVICE_MEMORY means insufficient device (HSM) memory literally.
There are a lot of reasons of such bugs. We can’t consider all branches. It’s possible to restrict some issues only. Answering your question there are three main common reasons of problems with memory in Cryptoki library:
Memory for crypto operations. Client of Cryptoki is responsible for allocating such memory, not Cryptoki library. For example client of Cryptoki library must allocate buffer for final result before invoking C_EncryptFinal. If buffer size is not enough then Cryptoki returns CKR_BUFFER_TOO_SMALL.
HSM memory. CKR_DEVICE_MEMORY points to this case but it's beyond control of most software developers.
Memory for internal service structures in Cryptoki library. For example when you open session the memory for this structure is allocated. When you stop encryption process and start decryption within the same session the mode for this session changes. Cryptoki library should support internal state between calls because it supports multi-part iterative operations. When switching from one kind of operation to another it should free previous structures and allocate new ones in memory like heap. If application developer has library sources or wants to help in finding error it worth to do following in this situation(for this particular incident assuming library erroneously reports CKR_DEVICE_MEMORY instead of CKR_HOST_MEMORY). Try to run program only for one kind of operation (say encryption). If it works without memory error for mentioned period of time then it’s possible that memory leaks occurres while changing operation types. But you says that:"one session for encryption and the other for decryption". It narrows the scope. Probably memory for storing the state for multi-part operation leaks. Monitor the amount of memory after several operations. If you don't use multi-part operations then most likely it's the case 2 because Cryptoki library in such circumstances shouldn't allocate any non-stack memory.
These estimates are only to illustrate general issues with memory in such libraries.
You mention here that you are not closing your sessions. If that is true, that is most probably the cause of the CKR_DEVICE_MEMORYs.
I have also this problem and year is 2020 :S
.Net Framework + Rest Api couple have this problem this time.
I'm using HSM for decrypt method. I have a login method interactive channel, and we need to make performance test. The service has an instance from Pkcs11
pkcs11 = new Pkcs11(hsmPath, true);
slot = GetUsableSlot(pkcs11);
TokenInfo tokenInfo = slot.GetTokenInfo();
session = slot.OpenSession(true);
session.Login(CKU.CKU_USER, userLoginPin);
secretKey = GenerateKey(session);
And this is the Decrypt method.
public byte[] Decrypt(byte[] encryptedTextByteArray)
{
Mechanism mechanism = new Mechanism(CKM.CKM_AES_ECB);
byte[] sourceData = encryptedTextByteArray;
byte[] decryptedData = null;
using (MemoryStream inputStream = new MemoryStream(sourceData), outputStream = new MemoryStream())
{
try
{
session.Decrypt(mechanism, secretKey, inputStream, outputStream, 4096);
}
catch (Pkcs11Exception ex)
{
throw;
}
decryptedData = outputStream.ToArray();
}
return decryptedData;
}
When I try to make performance test using Postman runner, there is no problem for one thread.
If I increase thread count, It appears these errors.
First error: CKR_OPERATION_ACTIVE
Next error: CKR_DEVICE_MEMORY
I tried these methods.
-For every request closed session. And also opened session for new request. But not succeeed. The same errors appeared. (Of course request and response time increased)
-For evey request closed the conenction. And also opened new connection for new request. The same errors appeared. (Of course request and response time increased)
Anyone helps me? :)
Related
Okay, so "async all the way down" is the mandate. But when is it problematic?
For example, if you have limited access to a resource, as in a DbConnection or a file, when do you stop using async methods in favor of synchronous?
Let's review the complexity of an asynchronous database call:
(Not putting .ConfigureAwait(false) for readability.)
// Step 1: Ok, no big deal, our connection is closed, let's open it and wait.
await connection.OpenAsync();
// Connection is open! Let's do some work.
// Step 2: Acquire a reader.
using(var reader = await command.ExecuteReaderAsync())
{
// Step 3: Start reading results.
while(await reader.ReadAsync())
{
// get the data.
}
}
Steps:
Should be reasonably innocuous and nothing to worry about.
But now we've acquired an open connection in a potentially limited connection pool. What if when waiting for step 2, other long running tasks are at the head of the line in the task scheduler?
Even worse now, we await with an open connection (and most likely added latency).
Aren't we holding open a connection longer than necessary? Isn't this an undesirable result? Wouldn't it be better to use synchronous methods to lessen the overall connection time, ultimately resulting in our data driven application performing better?
Of course I understand that async doesn't mean faster but async methods provide the opportunity for more total throughput. But as I've observed, there can definitely be weirdness when there are tasks scheduled in-between awaits that ultimately delay the operation, and essentially behave like blocking because of the limitations of the underlying resource.
[Note: this question is focused on ADO, but this also applies to file reads and writes.]
Hoping for some deeper insight. Thank you.
There are a few things to consider here:
Database connection pool limits, specifically the "Max Pool Size" which defaults to 100. The database connection pool has upper limit of the maximum number of connections. Besure to set "Max Pool Size=X" where X is the maximum number of database connections you want to have. This applies to either sync or async.
The thread pool settings. The thread pool will not add threads quickly if you load spikes. It will only add a new thread every 500ms or so. See MSDN Threading Guidelines from 2004 and The CLR Thread Pool 'Thread Injection' Algorithm. Here is a capture of the number of busy threads on one of my projects. The load spiked and requests were delayed due to lack of available threads to service the requests. The line increases as new threads were being added. Remember every thread required 1MB of memory for its stack. 1000 threads ~= 1GB of RAM just for threads.
The load characteristics of your project, relates to the thread pool.
The type of system you are providing, I will assume you are talking about a ASP.NET type app/api
The throughput (requests/sec) vs latency (sec/request) requirements. Async will add to latency but increase throughput.
The database/query performance, relates to the 50ms recommendation below
The article The overhead of async/await in NET 4.5 Edit 2018-04-16 the recommendation below applied to WinRT UI based applications.
Avoid using async/await for very short methods or having await
statements in tight loops (run the whole loop asynchronously instead).
Microsoft recommends that any method that might take longer than 50ms
to return should run asynchronously, so you may wish to use this
figure to determine whether it’s worth using the async/await pattern.
Also take a watch Diagnosing issues in ASP.NET Core Applications - David Fowler & Damian Edwards that talks about issues with thread pool and using async, sync, etc.
Hopefully this helps
if you have limited access to a resource, as in a DbConnection or a file, when do you stop using async methods in favor of synchronous?
You shouldn't need to switch to synchronous at all. Generally speaking, async only works if it's used all the way. Async-over-sync is an antipattern.
Consider the asynchronous code:
using (connection)
{
await connection.OpenAsync();
using(var reader = await command.ExecuteReaderAsync())
{
while(await reader.ReadAsync())
{
}
}
}
In this code, the connection is held open while the command is executed and the data is read. Anytime that the code is waiting on the database to respond, the calling thread is freed up to do other work.
Now consider the synchronous equivalent:
using (connection)
{
connection.Open();
using(var reader = command.ExecuteReader())
{
while(reader.Read())
{
}
}
}
In this code, the connection is held open while the command is executed and the data is read. Anytime that the code is waiting on the database to respond, the calling thread is blocked.
With both of these code blocks, the connection is held open while the command is executed and the data is read. The only difference is that with the async code, the calling thread is freed up to do other work.
What if when waiting for step 2, other long running tasks are at the head of the line in the task scheduler?
The time to deal with thread pool exhaustion is when you run into it. In the vast majority of scenarios, it isn't a problem and the default heuristics work fine.
This is particularly true if you use async everywhere and don't mix in blocking code.
For example, this code would be more problematic:
using (connection)
{
await connection.OpenAsync();
using(var reader = command.ExecuteReader())
{
while(reader.Read())
{
}
}
}
Now you have asynchronous code that, when it resumes, blocks a thread pool thread on I/O. Do that a lot, and you can end up in a thread pool exhaustion scenario.
Even worse now, we await with an open connection (and most likely added latency).
The added latency is miniscule. Like sub-millisecond (assuming no thread pool exhaustion). It's immeasurably small compared to random network fluctuations.
Aren't we holding open a connection longer than necessary? Isn't this an undesirable result? Wouldn't it be better to use synchronous methods to lessen the overall connection time, ultimately resulting in our data driven application performing better?
As noted above, synchronous code would hold the connection open just as long. (Well, OK, a sub-millisecond amount less, but that Doesn't Matter).
But as I've observed, there can definitely be weirdness when there are tasks scheduled in-between awaits that ultimately delay the operation, and essentially behave like blocking because of the limitations of the underlying resource.
It would be worrying if you observed this on the thread pool. That would mean you're already at thread pool exhaustion, and you should carefully review your code and remove blocking calls.
It's less worrying if you observed this on a single-thread scheduler (e.g., UI thread or ASP.NET Classic request context). In that case, you're not at thread pool exhaustion (though you still need to carefully review your code and remove blocking calls).
As a concluding note, it sounds as though you're trying to add async the hard way. It's harder to start at a higher level and work your way to a lower level. It's much easier to start at the lower level and work your way up. E.g., start with any I/O-bound APIs like DbConnection.Open / ExecuteReader / Read, and make those asynchronous first, and then let async grow up through your codebase.
Due to the way database connection pooling works at lower levels of protocol, the high level open / close commands don't have a lot of effect on performance. Generally though the internal thread scheduling IO is usually not a bottleneck unless you have some really long running tasks - we're talking something CPU intensive or worse - blocking inside. This will quickly exhaust your thread pool and things will start queuing up.
I would also suggest you investigate http://steeltoe.io, particularly the circuit breaker hystrix implementation. The way it works is it allows you to group your code into commands, and have command execution managed by command groups, which are essentially dedicated and segregated thread pools. The advantage is if you have a noisy, long running command, it can only exhaust it's own's command group thread pool without affecting the rest of the app. There are many other advantages of this portion of the library, primary being circuit breaker implementation, and one of my personal favorite collapsers. Imagine multiple incoming calls for a query GetObjectById being grouped into a single select * where id in(1,2,3) query and then results mapped back on the separate inbound requests. Db call is just an example, can be anything really.
Significant amounts of iteration introduce significant added latency and extra CPU usage
See http://telegra.ph/SqlDataReader-ReadAsync-vs-Read-04-18 for details.
As suspected:
Using async does not come without cost and requires consideration.
Certain types of operations lend themselves well to async, and others are problematic (for what should be obvious reasons).
High volume synchronous/blocking code has it's downsides, but for the most part is well managed by modern threading:
Testing / Profiling
4 x 100 paralleled queries, 1000 records each query.
Performance Profile for Synchronous Query
Average Query: 00:00:00.6731697, Total Time: 00:00:25.1435656
Performance Profile for Async Setup with Synchronous Read
Average Query: 00:00:01.4122918, Total Time: 00:00:30.2188467
Performance Profile for Fully Async Query
Average Query: 00:00:02.6879162, Total Time: 00:00:32.6702872
Assessment
The above results were run on SQL Server 2008 R2 using a .NET Core 2 console application. I invite anyone who has access to a modern instance of SQL Server to replicate these tests to see if there is a reversal in trend. If you find my testing method flawed, please comment so I correct and retest.
As you can easily see in the results. The more asynchronous operations we introduce, the longer the the queries take, and the longer the total time to complete. Even worse, fully asynchronous uses more CPU overhead which is counter productive to the idea that using async tasks would provide more available thread time. This overhead could be due to how I'm running these tests, but it's important to treat each test in a similar way to compare. Again, if anyone has a way to prove that async is better, please do.
I'm proposing here that "async all the way" has it's limitations and should be seriously scrutinized at certain iterative levels (like file, or data access).
I need to put a customized logging system of sorts in place for an ASP.NET application. Among other things, it has to log some data per request. I've thought of two approaches:
Approach #1: Commit each entry per request. For example: A log entry is created and committed to the database on every request (using a transient DbContext). I'm concerned that this commit puts an overhead on the serving of the request that would not scale well.
Approach #2: Buffer entries, commit periodically. For example: A log entry is created and added to a concurrent buffer on every request (using a shared lock). When a limit in that buffer is exceeded, an exclusive lock is acquired, the buffered entries are committed to the database in one go (using another, also transient DbContext, created and destroyed only for each commit) and the buffer is emptied. I'm aware that this would make the "committing" request slow, but it's acceptable. I'm also aware that closing/restarting the application could result in loss of uncommitted log entries because the AppDomain will change in that case, but this is also acceptable.
I've implemented both approaches within my requirements, I've tested them and I've strained them as much as I could in a local environment. I haven't deployed yet and thus I cannot test them in real conditions. Both seem to work equally well, but I can't draw any conclusions like this.
Which of these two approaches is the best? I'm concerned about performance during peaks of a couple thousand users. Are there any pitfalls I'm not aware of?
To solve your concern with option 1 about slowing down each request, why not use the TPL to offload the logging to a different thread? Something like this:
public class Logger
{
public static void Log(string message)
{
Task.Factory.StartNew(() => { SaveMessageToDB(message); });
}
private static void SaveMessageToDB(string message)
{
// etc.
}
}
The HTTP request thread wouldn't have to wait while the entry is written. You could also adapt option 2 to do the same sort of thing to write the accumulated set of messages in a different thread.
I implemented a solution that is similar to option 2, but in addition to a number limit, there was also a time limit. If no logs entries had been entered in a certain number of seconds, the queue would be dumped to the db.
Use log4net, and set its buffer size appropriately. Then you can go home and have a beer the rest of the day... I believe it's Apache licensed, which means you're free to modify/recompile it for your own needs (fitting whatever definition of "integrated in the application, not third party" you have in mind).
Seriously though - it seems way premature to optimize out a single DB insert per request at the cost of a lot of complexity. If you're doing 10+ log calls per request, it would probably make sense to buffer per-request - but that's vastly simpler and less error prone than writing high-performance multithreaded code.
Of course, as always, the real proof is in profiling - so fire up some tests, and get some numbers. At minimum, do a batch of straight inserts vs your buffered logger and determine what the difference is likely to be per-request so you can make a reasonable decision.
Intuitively, I don't think it'd be worth the complexity - but I have been wrong on performance before.
I have an asp.net app which uses legacy COM interop library. Works fine until memory reaches somewhere around 500Mb and then it is no longer able to create new COM objects (get various exceptions, e.g. Creating an instance of the COM component with CLSID {FFFF-FFFF-FFFF-FFF-FFFFFF} from the IClassFactory failed due to the following error: 80070008.). It almost looks like it is hitting some kind of memory limit, but what is it? Can it be changed?
Solved! Turns out the object was creating a Window handle and we were hitting the 10K Window handles limit (except it was happening at 2K instances for some reason when inside IIS)
Solved! Turns out the object was creating a Window handle and we were hitting the 10K Window handles limit (except it was happening at 2K instances for some reason when inside IIS)
What OS, and is it 32-bit or 64-bit? What are you using to determine memory usage?
When you say you're explicitly releasing the objects, do you mean you're using Marshal.ReleaseComObject()?
I'm assuming you have AspCompat=true in your <%# Page > tag... wouldn't expect it to run at all if you didn't.
Can you give us some details on your COM object; what does it do, and can you post some code where you're calling it, including COM object signatures? How much memory would you expect a single object to take?
My first suspect, based only on the information that I've read so far, is that 500Mb is not truly the total memory in use, and/or that you're having a memory fragmentation issue. I've seen this occur with IIS processes when less than half of the memory is in use, and the errors tend to be random, depending on what object is being created at the time. BTW, 80070008 is 'not enough storage space'.
Process limits are 2GB on a 32-bit machine, of course, but even if a process isn't using the full 2GB, if there's not a contiguous block of memory of the size needed when creating an object, you'll get an out-of-memory error when you try to allocate. Lots of concurrent users implies lots of COM objects (and other objects) being allocated and released in a short period of time... which points to fragmentation as a suspect.
Coming up with an attack plan requires more info about the COM object and how it's being used.
Use a command pattern for queueing and executing the com interop in an asynchronous thread. This can free up the number of threads being used by iis, and allow you to control the number of calls/instances of the com app.
You may think about object pooling rather than creating every time a new object.
I am calling a RESTful web service in the back-end of some ASP.NET pages.
I am using ASP.NET asynchronous pages, so under the hood I am using the methods:
HttpWebRequest BeginGetResponse()
and
HttpWebRequest EndGetResponse()
The response string in my case is always a JSON string. I use the following code to read the entire string:
using (StreamReader sr = new StreamReader(myHttpWebResponse.GetResponseStream()))
{
myObject.JSONData = sr.ReadToEnd();
}
Is this method OK in terms of scalability? I have seen other code samples that instead retrieve the response data in blocks using Read(). My primary goal is scalability, so this back-end call can be made across many concurrent page hits.
Thanks,
Frank
It depends on what you mean by "scalable". If you're talking about being able to handle bigger and bigger files, I'd say it's not terribly scalable. Since you're using a single ReadToEnd, a huge stream would require the entire stream be read into memory and then acted upon. As the application streams grow in number, complexity and size you're going to find that this will begin to hamper the server's performance to handle requests. You may also find that your application pool will begin to recycle itself DURING your request (if you end up taking that much virtual memory).
If the stream is always going to be smallish and you're only concerned with the number of streams created, I don't see why this wouldn't scale as long as your streams were dependent on open files, database connections, etc.
My huge 32-bit web services LLBLGen-based data access application is running alone on a dedicated 64-bit machine. Its physical memory consumption steadily grows up to approximately 2GB when the process releases almost all of the allocated space (up to 1,5GB) and continues to grow from that point again. There is no observable increase in Page Input values or other page file usage parameters so it looks like the memory is released rather than being swapped out to page file. I am thinking what kind of profile is this? There is nothing to actually prevent the process from grabbing all memory it can, on the other hand there are unacceptable http internal errors around the memory release - probably the clean-up blocks useful work. What would be a good strategy to make the cleanup less obtrusive, given the above is an acceptable behaviour in the first place.
It sounds like you have a memory leak, the process keeps leaking memory until it crushes with an out-of-memory condition and is then automatically restarted by the server.
1.5GB is about the maximum amount of memory a 32 bit process can allocate before running out of address space.
Somethings to look for:
Do you do your own caching? when are items removed from the cache?
Is there somewhere data is added to a collection every once in a while but never removed?
Do you call Dispose on every object that implements IDisposable?
Do you access any non-managed code at all (COM objects or using DllImport) or allocate non-managed memory (using the Marshal class for example)? anything that is allocated there is never freed by the garbage collector, you have to free it yourself.
Do you use 3rd party libraries or any code from 3rd parties? it can have any of the problems in the list too.
Is it possible you are not disposing of various disposable objects (particular DB related). This would leave them around, potentially tying up large amounts of unmanaged resources until the GC runs and their finalizers are called.
It would be worth running perfmon against you process and looking to see if there is a steady growth in some critical resource, like handles, or if your DB provider exposes performance counters then connections or open result sets.
I agree with the first part of edg's answer, but where he says:
"By setting objects to null when they
are dead you can encourage the GC to
reuse the memory consumed by those
objects, this limiting the growing
consumption of memory."
is incorrect. You never need to set an object to null since the GC will eventually collect your object after it goes out of scope.
This was discussed in this answer on SO: Setting Objects to Null/Nothing after use in .NET
Don't user Arraylists (garbage collect don't work weel with them), use instead generic lists
Other common error is to have in web.config Debug=true, this consume lot of memory, change the option to "false".
Other thing to do is use CLRProfiler, to trace the problem.
Good Luck,
Pedro
The Garbage Collector doesn't automatically free memory when it releases objects, it holds on to that memory to help minimise the expense of future mallocs.
When a low memory condition is triggered that memory will be returned to the OS and you will see more available memory when looking through task manager. This will normally happen about the 2GB mark, or 3GB if you use the relevant switch.
<contentious>
By setting objects to null when they are dead you can encourage the GC to reuse the memory consumed by those objects, this limiting the growing consumption of memory.
But which objects should you set to null? Big objects, large collections, frequently created objects.
</contentious>
EDIT: There is evidence to support the value of setting objects to null. See this for detail. Of course there is no need to set objects to null, the point is does it help memory management in any way?
EDIT: We need a recent benchmark if such a thing exists rather than continuing to opine.
Ensure that you aren't putting up a debug build of your project. There's a feature* that when you have a debug build, if you instantiate any object that contains the definition for an event, even if you don't raise the event, it will hold only a small piece of memory indefinitely. Over time, these small pieces of memory will eat away at your memory pool, until it eventually restarts the web process, and start again.
*I call this a feature (and not a bug) because it's been around since the beginning of .Net 2 (not present in .Net 1.1), and there's been no patch to fix it. The memory leak must be due to some feature needed when debugging.
We were having similar situations occur and altered all our database connections to use a try/catch/finally approach.
Try was used to execute code, catch for error collection, and finally to close all variables and database connections.
internal BECollection<ReportEntity> GetSomeReport()
{
Database db = DatabaseFactory.CreateDatabase();
BECollection<ReportEntity> _ind = new BECollection<ReportEntity>();
System.Data.Common.DbCommand dbc = db.GetStoredProcCommand("storedprocedure");
try
{
SqlDataReader reader = (SqlDataReader)db.ExecuteReader(dbc);
while (reader.Read())
{
//populate entity
}
}
catch (Exception ex)
{
Logging.LogMe(ex.Message.ToString(), "Error on SomeLayer/SomeReport", 1, 1);
return null;
}
finally
{
dbc.Connection.Close();
_ind = null;
}
return _ind;
}
My first guess would be a memory leak. My second guess would be that it is normal behavior - the GC won't be fired until you have significant memory pressure. The only way to be sure is to use a combination of a profiler and things like PerfMon. Some sites:
http://blogs.msdn.com/ricom/archive/2004/12/10/279612.aspx
http://support.microsoft.com/kb/318263
Tess's excellent lab series
In addition I would make sure you aren't running in Debug mode (as already mentioned).
As far as the HTTP errors - assuming you are running in server GC mode, it tries to do everything it can to not block requests. It would be interesting to find out what those HTTP errors are - that's not normal behavior from what I've seen in the past, and might point to some more of the root of your issue.