Anybody knows if there is an option to set the RavenDB's storage initial size for the ESENT?
For VORON it should be, according to this site - Raven/Voron/InitialSize.
Thanks
There are a few settings that control this:
Raven/Esent/DbExtensionSize - By how much the db will grow, and its initial size.
Raven/Esent/LogFileSize - The size of the transaction logs
Related
My Java application's codecache is nearly full, so I use VM.getVM().getCodeCache().iterate() dump codecache info. Then call CodeBlob.getSize() count total size.
However codeBolbs' total size is not equals to JVM
How to get sun.jvm.hotspot.code.CodeBlob real size in codecache
I am trying to figure out why I am getting this error when indexing a document from a python web app.
The document in this case is a base64 encoded string of a file of size 10877 KB.
I post it to my web app, which then posts it via elasticsearch.py to my elastic instance.
My elastic instance throws an error:
TransportError(429, 'circuit_breaking_exception', '[parent] Data
too large, data for [<http_request>] would be
[1031753160/983.9mb], which is larger than the limit of
[986932838/941.2mb], real usage: [1002052432/955.6mb], new bytes
reserved: [29700728/28.3mb], usages [request=0/0b,
fielddata=0/0b, in_flight_requests=29700728/28.3mb,
accounting=202042/197.3kb]')
I am trying to understand why my 10877 KB file ends up at a size of 983mb as reported by elastic.
I understand that increasing the JVM max heap size may allow me to send bigger files, but I am more wondering why it appears the request size is 10x the size of what I am expecting.
Let us see what we have here, step by step:
[parent] Data too large, data for [<http_request>]
gives the name of the circuit breaker
would be [1031753160/983.9mb],
says, how the heap size will look, when the request would be executed
which is larger than the limit of [986932838/941.2mb],
tells us the current setting of the circuit breaker above
real usage: [1002052432/955.6mb],
this is the real usage of the heap
new bytes reserved: [29700728/28.3mb],
actually an estimatiom, what impact the request will have (the size of the data structures which needs to be created in order to process the request). Your ~10MB file will probably consume 28.3MB.
usages [
request=0/0b,
fielddata=0/0b,
in_flight_requests=29700728/28.3mb,
accounting=202042/197.3kb
]
This last line tells us how the estmation is being calculated.
I've been looking for a authoritative source of azure cosmosdb limits by I can't find one. In particular, I need to know the size limits for a individual item.
The maximum size of a document today is 2MB.
https://learn.microsoft.com/en-us/azure/cosmos-db/documentdb-resources#documents
So this is one of those things that always annoys me about documentation.
Sure it's 2MB, but by who's measuring stick.
TLDR: Between 2,090,014 and 2,100,014 when
Encoding.UTF8.GetByteCount(doc) or
Encoding.ASCII.GetByteCount(doc)
To get there I set up the following code:
for (int i = 10; i < 10000; i++)
{
var docItem = new TestItem(new string('A', i * 10000));
string doc = JsonConvert.SerializeObject(docItem);
log.LogInformation(" ");
log.LogInformation(" -------------------------------------------------------------------------------------------------------------------------------------");
log.LogInformation($" ------------------------------------------------- Doc Size = {i*10000 } --------------------------------------------------");
log.LogInformation(" -------------------------------------------------------------------------------------------------------------------------------------");
log.LogWarning($"UTF7 - {Encoding.UTF7.GetByteCount(doc)}");
log.LogWarning($"UTF8 - {Encoding.UTF8.GetByteCount(doc)}");
log.LogWarning($"UTF32 - {Encoding.UTF32.GetByteCount(doc)}");
log.LogWarning($"Unicode - {Encoding.Unicode.GetByteCount(doc)}");
log.LogWarning($"Ascii - {Encoding.ASCII.GetByteCount(doc)}");
log.LogInformation(" -------------------------------------------------------------------------------------------------------------------------------------");
log.LogWarning($"UTF7 - {ASCIIEncoding.UTF7.GetByteCount(doc)}");
log.LogWarning($"UTF8 - {ASCIIEncoding.UTF8.GetByteCount(doc)}");
log.LogWarning($"UTF32 - {ASCIIEncoding.UTF32.GetByteCount(doc)}");
log.LogWarning($"Unicode - {ASCIIEncoding.Unicode.GetByteCount(doc)}");
log.LogWarning($"Ascii - {ASCIIEncoding.ASCII.GetByteCount(doc)}");
try
{
await cosmosStore.CreateDocumentAsync(docItem);
}
catch (Exception e)
{
log.LogWarning(e.Message + "Caught");
}
}
And here's where it broke:
Update: increasing the max size to 16 MB is now possible.
https://devblogs.microsoft.com/cosmosdb/larger-document-sizes-unique-index-improvements-expr-support-in-azure-cosmos-db-api-for-mongodb/
The max allowable document size is 2 MB. This is fixed for Azure Cosmos DB for NOSQL API account
If it Exceeds you face 413 error --> 413 Entity too large The document size in the request exceeded the allowable document size for a request.
If your Environment is already in Production
we still suggest reducing the document size as a solution.
You can Reduce the document size/ Re model your data
More Info:
Azure Cosmos DB service quotas | Microsoft Learn
https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-item-limits
Additional Information:
For Azure Cosmos DB for API for MongoDB
If your Azure Cosmos DB account is Mongo DB API , the limit is 2 MB but
There is a preview feature and please be noted that this is not recommended for prod environment:
There is a preview feature to set 16MB limit per document in API for MongoDB.
Ref:
https://azure.microsoft.com/en-us/updates/public-preview-16mb-limit-per-document-in-api-for-mongodb/
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/feature-support-42#data-types
However, it’s in preview and not recommended for Production environments. There’s no ETA when it will be GA.
var commentsRef = new Firebase('https://test.firebaseio.com/comments');
var last10Comments = commentsRef.limit(10);
//Rendering last 10 comments
last10Comments.on('child_added', function (snapshot) {
});
From the client side a user can change the limit number and can render all comments from comments reference.
Is there any way to restrict reading limit to some number at any point of time for a location?
No, there isn't currently a way to put Firebase security rules around that type of limiting of data. Another approach that would work would be to have another section of the tree that contains a denormalized portion of the data that just contains the last 10 comments and nothing more.
Thanks for bringing this up. I've added this to our internal tracker to keep it in mind when we design V2 of our security API.
Here I have a simple feature on ASP.NET MVC3 which host on Azure.
1st step: user upload a picture
2nd step: user crop the uploaded picture
3rd: system save the cropped picture, delete the temp file which is the uploaded original picture
Here is the problem I am facing now: where to store the temp file?
I tried on windows system somewhere, or on LocalResources: the problem is these resources are per Instance, so here is no guarantee the code on an instance shows the picture to crop will be the same code on the same instance that saved the temp file.
Do you have any idea on this temp file issue?
normally the file exist just for a while before delete it
the temp file needs to be Instance independent
Better the file can have some expire setting (for example, 1H) to delete itself, in case code crashed somewhere.
OK. So what you're after is basically somthing that is shared storage but expires. Amazon have just announced a rather nice setting called object expiration (https://forums.aws.amazon.com/ann.jspa?annID=1303). Nothing like this for Windows Azure storage yet unfortunately, but, doesnt mean we can't come up with some other approach; indeed even come up with a better (more cost effective) approach.
You say that it needs to be instance independant which means using a local temp drive is out of the picture. As others have said my initial leaning would be towards Blob storage but you will have cleanup effort there. If you are working with large images (>1MB) or low throughput (<100rps) then I think Blob storage is the only option. If you are working with smaller images AND high throughput then the transaction costs for blob storage will start to really add up (I have a white paper coming out soon which shows some modelling of this but some quick thoughts are below).
For a scenario with small images and high throughput a better option might be to use the Windows Azure Cache as your temporary storaage area. At first glance it will be eye wateringly expensive; on a per GB basis (110GB/month for Cache, 12c/GB for Storage). But, with storage your transactions are paid for whereas with Cache they are 'free'. (Quotas are here: http://msdn.microsoft.com/en-us/library/hh697522.aspx#C_BKMK_FAQ8) This can really add up; e.g. using 100kb temp files held for 20 minutes with a system throughput of 1500rps using Cache is about $1000 per month vs $15000 per month for storage transactions.
The Azure Cache approach is well worth considering, but, to be sure it is the 'best' approach I'd really want to know;
Size of images
Throughput per hour
A bit more detail on the actual client interaction with the server during the crop process? Is it an interactive process where the user will pull the iamge into their browser and crop visually? Or is it just a simple crop?
Here is what I see as a possible approach:
user upload the picture
your code saves it to a blob and have some data backend to know the relation between user session and uploaded image (mark it as temp image)
display the image in the cropping user interface interface
when user is done cropping on the client:
4.1. retrieve the original from the blob
4.2. crop it according the data sent from the user
4.3. delete the original from the blob and the record in the data backend used in step 2
4.4. save the final to another blob (final blob).
And have one background process checking for "expired" temp images in the data backend (used in step 2) to delete the images and the records in the data backend.
Please note that even in WebRole, you still have the RoleEntryPoint descendant, and you still can override the Run method. Impleneting the infinite loop in the Run() (that method shall never exit!) method, you can check if there is anything for deleting every N seconds (depending on your Thread.Sleep() in the Run().
You can use the Azure blob storage. Have look at this tutorial.
Under sample will be help you.
https://code.msdn.microsoft.com/How-to-store-temp-files-in-d33bbb10
you have two way of temp file in Azure.
1, you can use Path.GetTempPath and Path.GetTempFilename() functions for the temp file name
2, you can use Azure blob to simulate it.
private long TotalLimitSizeOfTempFiles = 100 * 1024 * 1024;
private async Task SaveTempFile(string fileName, long contentLenght, Stream inputStream)
{
try
{
//firstly, we need check the container if exists or not. And if not, we need to create one.
await container.CreateIfNotExistsAsync();
//init a blobReference
CloudBlockBlob tempFileBlob = container.GetBlockBlobReference(fileName);
//if the blobReference is exists, delete the old blob
tempFileBlob.DeleteIfExists();
//check the count of blob if over limit or not, if yes, clear them.
await CleanStorageIfReachLimit(contentLenght);
//and upload the new file in this
tempFileBlob.UploadFromStream(inputStream);
}
catch (Exception ex)
{
if (ex.InnerException != null)
{
throw ex.InnerException;
}
else
{
throw ex;
}
}
}
//check the count of blob if over limit or not, if yes, clear them.
private async Task CleanStorageIfReachLimit(long newFileLength)
{
List<CloudBlob> blobs = container.ListBlobs()
.OfType<CloudBlob>()
.OrderBy(m => m.Properties.LastModified)
.ToList();
//get total size of all blobs.
long totalSize = blobs.Sum(m => m.Properties.Length);
//calculate out the real limit size of before upload
long realLimetSize = TotalLimitSizeOfTempFiles - newFileLength;
//delete all,when the free size is enough, break this loop,and stop delete blob anymore
foreach (CloudBlob item in blobs)
{
if (totalSize <= realLimetSize)
{
break;
}
await item.DeleteIfExistsAsync();
totalSize -= item.Properties.Length;
}
}