I incresed the cache for my squid proxy. But i really do not understand the params before the size-MB param.
The reference show the fallow:
cache_dir scheme directory size-MB L1 L2
What is L1 and L2 params and how calculate the proper value?
Configuring the number of sub directories
There are two arguments to cache_dir named as L1 and L2. Squid stores the cacheable objects in a hierarchical fashion in directories named so that it'll be faster to lookup an object in the cache. The hierarchy is of two-levels, where L1 determines the number of directories at the first level and L2 determines the number of directories in each of the directories at
the first level. We should set L1 and L2 high enough so that directories at the second level don't have a huge number of files.
Let's see an example of the cache directory /squid_cache/ with 50GB of
free space:
cache_dir aufs /squid_cache/ 51200 32 512
We have a cache directory /squid_cache/ with 50 GB of free space with the values of L1 and L2 as 32 and 512 respectively. So, if we assume the average size of a cached object to be 16 KB, there will be 51200x1024÷(32x512x16) = 200 cached objects in each of the directories at the second level, which is quite good.
Read the Book Squid3 Beginners Guide
Related
I have a FAT16 drive that contains the following info:
Bytes per sector: 512 bytes (0x200)
Sectors per cluster: 64 (0x40)
Reserved sectors: 6 (0x06)
Number of FATs: 2 (0x02)
Number of root entries: 512 (0x0200)
Total number of sectors: 3805043 (0x3a0f73)
Sectors per file allocation table: 233 (0xE9)
Root directory is located at sector 472 (0x1d8)
I'm looking for a file with the following details:
File name: LOREMI~1
File extension: TXT
File size: 3284 bytes (0x0cd4)
First cluster: 660 (0x294)
However, I would admit that the start of the file cluster is located at sector 42616. My problem is that what equation should I use that would produce 42616?
I have trouble figuring this out since there is barely any information about this other than a tutorial made by Tavi Systems but the part involving this is very hard to follow.
Actually, the FAT filesystem is fairly well documented. The official FAT documentation by Microsoft can be found by the filename fatgen103.
The directory entry LOREMI~1.TXT can be found in the root directory and is precedented by the long file name entry (xt, lorem ipsum.t → lorem ipsum.txt), the directory entry is documented in the «FAT Directory Structure» chapter; in case of FAT16 you are interested in the 26th to 28th byte to get the cluster address (DIR_FstClusLo), which is (little endian!) 0x0294 (or 660₁₀).
Based on the BPB header information you provided we can calculate the the data sector like this:
data_sector = (cluster-2) * sectors_per_cluster +
(reserved_sectors + (number_of_fats * fat_size) +
first_data_sector)
Why cluster-2? Because the first two clusters in a FAT filesystem are always reserved for the BPB header block as well as the FAT itself, see chapter «FAT Data Structure» in fatgen103.doc.
In order for us to solve this, we still need to determine the sector span of the root directory entry. For FAT12/16 this can be determined like this:
first_data_sector = ((root_entries * directory_entry_size) +
(bytes_per_sector - 1)) // bytes_per_sector
The directory entry size is always 32 bytes as per specification (see chapter «FAT Directory Structure» in fatgen103.doc), every other value is known by now:
first_data_sector = ((512*32)+(512-1)) // 512 → 32
data_sector = (660-2)*64+(6+(2*233)+32) → 42616
I am trying to stress a ubuntu container's memory. Typing free in my command terminal provides the following result:
free -m
total used free shared buff/cache available
Mem: 7958 585 6246 401 1126 6743
Swap: 2048 0 2048
I want to stress exactly 10% of the total available memory. Per stress-ng manual:
-m N, --vm N
start N workers continuously calling mmap(2)/munmap(2) and writing to the allocated
memory. Note that this can cause systems to trip the kernel OOM killer on Linux
systems if not enough physical memory and swap is not available.
--vm-bytes N
mmap N bytes per vm worker, the default is 256MB. One can specify the size as % of
total available memory or in units of Bytes, KBytes, MBytes and GBytes using the
suffix b, k, m or g.
Now, on my target container I run two memory stressors to occupy 10% of my memory:
stress-ng -vm 2 --vm-bytes 10% -t 10
However, the memory usage on the container never reaches 10% no matter how many times I run it. I tried different timeout values, no result. The closet it gets is 8.9% never approaches 10%. I inspect memory usage on my container this way:
docker stats --no-stream kind_sinoussi
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
c3fc7a103929 kind_sinoussi 199.01% 638.4MiB / 7.772GiB 8.02% 1.45kB / 0B 0B / 0B 7
In an attempt to understand this behaviour, I tried running the same command with an exact unit of bytes. In my case, I'll opt for 800 mega since 7958m * 0.1 = 795,8 ~ 800m.
stress-ng -vm 2 --vm-bytes 800m -t 15
And, I get 10%!
docker stats --no-stream kind_sinoussi
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
c3fc7a103929 kind_sinoussi 198.51% 815.2MiB / 7.772GiB 10.24% 1.45kB / 0B 0B / 0B 7
Can someone explain why this is happening?
Another question, is it possible for stress-ng to stress memory usage to 100%?
stress-ng --vm-bytes 10% will use sysconf(_SC_AVPHYS_PAGES) to determine the available memory. This sysconf() system call will return the number of pages that the application can use without hindering any other process. So this is approximately what the free command is returning for the free memory statistic.
Note that stress-ng will allocate the memory with mmap, so it may be that during run time mmap'd pages may not necessarily be physically backed at the time you check how much real memory is being used.
It may be worth trying to also use the --vm-populate option; this will try and ensure the pages are physically populated on the mmap'd memory that stress-ng is exercising. Also try --vm-madvise willneed to use the madvise() system call to hint that the pages will be required fairly soon.
I'm trying to compare in java a huge amount of data with 2 folders entry folder1 and folder2.
Each folder contains several files of around 10 MBytes size each. I have about 100 files in each folder.
Each file contains a key,value line something like (total is around 500 Millions of lines for each folder):
RFE023334343432-45,456677
RFE54667565765-5,465368
and son on..
First Step
At first, each line of all files from folder1 is read and load into a rocksdb using
in my example above
key = RFE023334343432-45 and corresponding
value = 456677
Second step
Once my rocksdb is full with folder1 data,
for each line read in folder2, I call the folder1 rocksdb get() method to check if the key extracted for folder2 line exists into the rocksdb. It returns null when I does not exist.
Note that I cannot use rocksdb keyMayExist() method because It returns false positive results when you manipulate a too huge amount of data.
The performance are correct when the data inside the folder1 is ordered regarding the key value.
But my duration is multiplied by 3 when the input data are not sorted (I shuffled them using shell command). That is weird because in my test
I copy my unsort folder1 into folder2 (just duplication my folder). Thus even if folder1 is unsorted, folder2 is also unsorted in exactly the same way as folder1.
My question is how can I sort my rocksdb by key?
RocksDB always sorts data by key. You can use an iterator to K/V pairs from a RocksDB instance. Here is the API to create an iterator: https://github.com/facebook/rocksdb/blob/v6.22.1/include/rocksdb/db.h#L709-L716
I think this might be a quite common problem, but somehow it is hard to find suitable answers.
Background
I'm currently investigating how to speed up the node variant of Patternlab. It is basically a system where you create HTML template files containing some fundamental elements (called atoms) like headings, paragraphs, etc. and compose them into something bigger (molecules), then compose multiple molecules until you have a full web page.
The template files are processed by a template engine which is able to reference and include other template files.
Patternlab now would read all files and build a graph of each file's predecessors and ancestors.
The problem is, all files in the template directory(s) are read, processed, the compile output from the template engine is written somewhere, done. This takes about 1 Minute for 10-20 MB of HTML Templates (think e.g. of Mustache). It would be much nicer to have differential updates.
General problem
Given a directed acyclic graph of files(nodes) f and f1 → f2 for f1 includes f2 in its contents. When in f1 → f2 needs also to be recompiled(i.e. the compilation order should be f2, f1) , how to find the set of all files that need recompilation efficiently and what are the initial and target nodes, when there might be a path containing multiple changed files and nodes depending on multiple paths?
Given two parallel paths (i.e. f1 → f2 → f5 and f1 → f4 → f5) how to find the best set of paths for parallel compilation (i.e. by total length of the path) with a minimum length l?
Considerations
So how could this be done? We simply have to watch the template directories for the events create, modify and delete. This could be done by collecting file system events from the kernel.
Remember that for any file we can also receive the "last modified" timestamp (mtime).
The first step, building an initial graph is already done by patternlab. Each node of the graph consists of:
the source file path and mtime
the target file path and mtime
a set of parameters for the file (i.e. username, page title, etc.)
a changed flag that indicates if the file was modified by inspecting the mtimes(see below). This always reflects the current file system state (much simpler).
i.e. there is a "shadow" graph compiling the most current compile output.
This leads to these cases:
source.mtime <= target.mtime: The source was compiled and not modified afterwards. If the source file is changed after building the graph,
source.mtime > target.mtime: The source was changed before building the graph and must be recompiled
Now the idea is to recompile each node if the changed flag is set and return the contents of the target file otherwise.
When a file is compiled, all files referencing this file (e.g. all nodes with incoming edges to the node) must be recompiled as well.
Consider that when the source hsa been compiled to the target, the flag changes to false so it does not get compiled twice. Returning the target content when visited again is okay though.
There are multiple root nodes (only outgoing edges, i.e. pages),some that are "leafs" (only incoming edges, i.e. molecules) and also nodes with no
Assume that patternlab cannot run in paralell, i.e. there is a "busy" flag blocking concurrent overall runs.
However it might be beneficial to use all CPU cores for compiling templates faster and employ content caching for templates that were not modified.
If a file f1 includes two other files f2, f3 and f2, f3 both have files that need recompilations as successors, f1 must wait for all paths to be recompiled first.
For instance (c = changed)
f5 -> f6 (c)
^
/
f1 -> f2 -> f3 -> f4(c)
I am using a 32 bit perl in my openvms system.(So perl can access up till 2gb of virtual address space ).
I am hitting "out of memory!" in a large perl script. I zeroed in on the location of variable causing this . However after my tests with devel:size it turns out the array is using only 13 Mb memory and the hash is using much less than that.
My question is about memory profiling this perl script in VMS.
is there a good way of doing memory profile on VMS?
I used size to get size of array and hash.(Array is local scope and hash is global scope)
DV Z01 A4:[INTRO_DIR]$ perl scanner_SCANDIR.PL
Directory is Z3:[new_dir]
13399796 is total on array
3475702 is total on hash
Directory is Z3:[new_dir.subdir]
2506647 is total on array
4055817 is total on hash
Directory is Z3:[new_dir.subdir.OBJECT]
5704387 is total on array
6040449 is total on hash
Directory is Z3:[new_dir.subdir.XFET]
1585226 is total on array
6390119 is total on hash
Directory is Z3:[new_dir.subdir.1]
3527966 is total on array
7426150 is total on hash
Directory is Z3:[new_dir.subdir.2]
1698678 is total on array
7777489 is total on hash
(edited: Pmis-spelled GFLQUOTA )
Where is that output coming from? To OpenVMS folks it suggests files in directories, which the code might suck in? There would typically be considerable malloc/align overhead per element saved.
Anyway the available ADDRESSABLE memory when strictly using 32 pointers on OpenVMS is 1GB: 0x0 .. 0x3fffffff, not 2GB, for programs and (malloc) data for 'P0' space. There is also room in P1 (0x7fffffff .. 0x4000000) for thread-local stack storages, but perl does not use (much) of that.
From a second session you can look at that with DCL:
$ pid = "xxxxxxxx"
$ write sys$output f$getjpi(pid,"FREP0VA"), " ", f$getjpi(pid,"FREP1VA")
$ write sys$output f$getjpi(pid,"PPGCNT"), " ", f$getjpi(pid,"GPGCNT")
$ write sys$output f$getjpi(pid,"PGFLQUOTA")
However... those are just addresses ranges, NOT how much memory the process is allowed to used. That's governed by the process page-file-quota. Check with $ SHOW PROC/QUOTA before running perl. And its usage can be reported as per above from the outside adding Private pages and Groups-shared pages as per above.
An other nice way to look at memory (and other quota) is SHOW PROC/CONT ... and then hit "q"
So how many elements are stored in each large active array? How large is an average element, rounded up to 16 bytes? How many hash elements? How large are the key + value on average (round up generously)
What is the exact message?
Does the program 'blow' up right away, or after a while (so you can use SHOW PROC/CONT)
Is there a source file data set (size) that does work?
Cheers,
Hein.