I am trying to monitor the memory usage of kamailio. The PKG memory I have arranged, though the shared memory is something else.
According to documentation you can get the shared memory like this:
kamcmd mod.stats all shm
In my case, I get the following:
Module: core
{
msg_lump_cloner(984): 1560
dns_cache_mk_bad_entry(867): 192
dns_cache_mk_rd_entry(1216): 232
build_req_buf_from_sip_req(2155): 2752
sip_msg_shm_clone(495): 822664
tcpconn_new(957): 16416728
counters_prefork_init(207): 30720
cfg_clone_str(130): 208
cfg_shmize(217): 976
init_pt(105): 4200
cfg_parse_str(906): 80
init_pt(106): 16
init_pt(111): 40
create_avp(175): 10368
register_timer(1012): 720
cfg_register_ctx(47): 128
init_tcp(4643): 8192
init_tcp(4637): 32768
init_tcp(4629): 8
init_tcp(4622): 8
init_tcp(4615): 8
init_tcp(4609): 8
init_tcp(4597): 8
init_avps(90): 8
init_avps(89): 8
init_dst_blacklist(437): 16384
init_dst_blacklist(430): 8
timer_alloc(515): 96
init_dns_cache(366): 8
init_dns_cache(358): 16384
init_dns_cache(351): 16
init_dns_cache(345): 8
init_timer(284): 8
init_timer(283): 16384
init_timer(282): 8
init_timer(281): 8
init_timer(270): 8
init_timer(238): 8
init_timer(221): 278544
init_timer(220): 8
init_timer(207): 8
cfg_child_cb_new(830): 64
sr_cfg_init(361): 8
sr_cfg_init(354): 8
sr_cfg_init(347): 8
sr_cfg_init(335): 8
sr_cfg_init(323): 24
shm_core_lock_init(153): 8
Total: 17660616
}
... [ list of modules here, snipped because not relevant I think] ...
So I guess the one I need is Total: 17660616.
However, what does this mean? Is it the total allocated memory? Available memory?
The commands you used is for seeing the used of shared memory per module.
If you want the statistics for total used and free shared memory, then use the command:
kamctl stats shmem
or:
kamcmd stats.get_statistics shmem:
The output will be like:
shmem:fragments = 179
shmem:free_size = 127471848
shmem:max_used_size = 7269016
shmem:real_used_size = 6745880
shmem:total_size = 134217728
shmem:used_size = 5106728
The free_size value is the one you have to monitor for available shared memory.
Related
I am in the early stages of building/testing my own defined dictionary. I am testing it with a set of American state party platforms (corpus of 30 txt files). I have successfully created the dictionary and used Quanteda to provide summary statistics, but it only seems to do this for 6 files at time and my plan is to use the dictionary on hundreds of files going back decades. Is there a way to display more than 6 documents at a time?
Here is the code I used that produced data frame for the 6 files and the error message:
corp_platform <- corpus(corp)
toks_platform <- tokens(corp_platform)
dict_toks <- tokens_lookup(toks_platform, dictionary = dict)
print(dict_toks)
dfm(dict_toks)
Document-feature matrix of: 30 documents, 2 features (1.67% sparse) and 2 docvars.
commmunitarian individualist
akdem20.txt 113 20
azdem20.txt 60 13
cadem20.txt 254 98
medem20.txt 27 7
mndfl20.txt 40 18
ncdem20.txt 235 64
[ reached max_ndoc ... 24 more documents ]
The print methods for core objects, such as dfm objects, by default only print a specified number of rows. That's what you are seeing here, and why it states:
Document-feature matrix of: 30 documents
[...]
.. 24 more documents ]
It's telling you that all 30 documents are there.
This is all documented. See help("print-methods", package = "quanteda"). If you want summary statistics, try quanteda.textstats::textstat_frequency(). Or if you want the dfm as a data.frame, use convert(dfm(dict_toks), to = "data.frame").
Thanks very much. I just needed a way to display the output, and could not find an example, so this is very helpful. I changed the code I have used to:
'corp_platform <- corpus(corp)
toks_platform <- quanteda::tokens(corp_platform)
dict_toks <- tokens_lookup(toks_platform, dictionary = dict)
print(dict_toks)
dfm(dict_toks)
convert(dfm(dict_toks), to = "data.frame")'
and the output is:
'doc_ id commmunitarian individualist
1 akdem20.txt 113 20
2 azdem20.txt 60 13
3 cadem20.txt 254 98
4 medem20.txt 27 7
5 mndfl20.txt 40 18
.........................................
25 tx2022draft.txt 198 156
26 txgop20.txt 181 153
27 wagop20.txt 52 63
28 wigop20.txt 27 11
29 wvgop20.txt 72 47
30 wygop20.txt 22 21'
I have compiled Quantum ESPRESSO (Program PWSCF v.6.7MaX) for GPU acceleration (hibrid MPI/OpenMP) with the next options:
module load compiler/intel/2020.1
module load hpc_sdk/20.9
./configure F90=pgf90 CC=pgcc MPIF90=mpif90 --with-cuda=yes --enable-cuda-env-check=no --with-cuda-runtime=11.0 --with-cuda-cc=70 --enable-openmp BLAS_LIBS='-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core'
make -j8 pw
Apparently, the compilation ends succesfully. Then, I execute the program:
export OMP_NUM_THREADS=1
mpirun -n 2 /home/my_user/q-e-gpu-qe-gpu-6.7/bin/pw.x < silverslab32.in > silver4.out
Then, the program starts running and print out the next info:
Parallel version (MPI & OpenMP), running on 8 processor cores
Number of MPI processes: 2
Threads/MPI process: 4
...
GPU acceleration is ACTIVE
...
Estimated max dynamical RAM per process > 13.87 GB
Estimated total dynamical RAM > 27.75 GB
But after 2 minutes of execution the job ends with error:
0: ALLOCATE: 4345479360 bytes requested; status = 2(out of memory)
0: ALLOCATE: 4345482096 bytes requested; status = 2(out of memory)
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[47946,1],1]
Exit code: 127
--------------------------------------------------------------------------
This node has > 180GB of available RAM. I check the Memory use with the top command:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
89681 my_user 20 0 30.1g 3.6g 2.1g R 100.0 1.9 1:39.45 pw.x
89682 my_user 20 0 29.8g 3.2g 2.0g R 100.0 1.7 1:39.30 pw.x
I noticed that the process stops when RES memory reaches 4GB. This are the caracteristics of the node:
(base) [my_user#gpu001]$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 28 29 30 31 32 33 34 35 36 37 38 39 40 41
node 0 size: 95313 MB
node 0 free: 41972 MB
node 1 cpus: 14 15 16 17 18 19 20 21 22 23 24 25 26 27 42 43 44 45 46 47 48 49 50 51 52 53 54 55
node 1 size: 96746 MB
node 1 free: 70751 MB
node distances:
node 0 1
0: 10 21
1: 21 10
(base) [my_user#gpu001]$ free -lm
total used free shared buff/cache available
Mem: 192059 2561 112716 260 76781 188505
Low: 192059 79342 112716
High: 0 0 0
Swap: 8191 0 8191
The version of MPI is:
mpirun (Open MPI) 3.1.5
This node is a compute node in a cluster, but no matter if I submit the job with SLURM or run it directly on the node, the error is the same.
Note that I compile it on the login node and run it on this GPU node, the difference is that on the login node it has no GPU connected.
I would really appreciate it if you could help me figure out what could be going on.
Thank you in advance!
I have just built a proof of concept for an asp.net MVC controller to (1) generate a barcode from user input using barcode rendering framework and (2) embed it in a PDF document using wkhtmltopdf.exe
Before telling my client it's a working solution, I want to know it's not going to bring down their website. My main concern is long-term reliability -- whether for instance creating and disposing the unmanaged system process for wkhtmltopdf.exe might leak something. (Peak performance and load is not expected to be such an issue - only a few requests per minute at peak).
So, I run a couple of tests from the Windows command line:
(1) 1,000 Requests in Sequence (ie 1 at a time)
for /l %i in (1,1,1000) do curl ^
"http://localhost:8003/Home/Test?text=Iteration_%i___012345&scale=2&height=50" -o output.pdf
(2) Up to 40 requests sent within 2 seconds
for /l %i in (1,1,40) do start "Curl %i" curl ^
"http://localhost:8003/Home/Test?text=Iteration_%i___012345&scale=2&height=50" -o output%i.pdf
And I record some performance counters in perfmon before, during & after. Specifically I look at total processes, threads, handles, memory, disk use on the machine and on the IIS process specifically.
So, my questions:
1) What would you consider acceptable evidence that the the solution looks to be at low risk of bringing down the server? Would you amend what I've done, or would you do something completely different?
2) Given my concern is reliability, I think that the 'Before' vs 'After' figures are the ones I most care about. Agree or not?
3) Looking at the Before vs After figures, the only concern I see is the 'Processes Total Handle Count'. I conclude that launching wkhtmltopdf.exe nearly a thousand times has probably not leaked anything or destabilised the machine. But I might be wrong and should run the same tests for hours or days to reduce the level of doubt. Agree or not?
(The risk level: A couple of people's jobs might be on the line if it went pear-shaped. Revenue on the site is £1,000s per hour).
My perfmon results were as follows.
700 Requests in Sequence
1-5 Mins 10 Mins
Counter Before Test Min Ave Max After Test
System
System Processes 95 97 100 101 95
System Threads 1220 1245 1264 1281 1238
Memory Available MB 4888 4840 4850 4868 4837
Memory % Committed 23 24 24 24 24
Processes Total Handle Cou 33255 34147 34489 34775 34029
Processor % Processor Time 4 to 30 40 57 78 1 to 30
Physical Disk % Disk Time 1 0 7 75 0 to 30
IIS Express
% Processor Time 0 0 2 6 0
Handle Count 610 595 640 690 614
Thread Count 34 35 35 35 35
Working Set 138 139 139 139 139
IO Data KB/sec 0 453 491 691 0
20 Requests sent within 2 seconds followed by 40 Requests sent within 3 seconds
1-5 Mins 10 Mins
Counter Before Test Min Ave Max After Test
System
System Processes 95 98 137 257 96
System Threads 1238 1251 1425 1913 1240
Memory Available MB 4837 4309 4694 4818 4811
Memory % Committed 24 24 25 29 24
Processes Total Handle Cou 34029 34953 38539 52140 34800
Processor % Processor Time 1 to 30 1 48 100 1 to 10
Physical Disk % Disk Time 0 to 30 0 7 136 0 to 10
IIS Express
% Processor Time 0 0 1 29 0
Handle Count 610 664 818 936 834
Thread Count 34 37 50 68 37
Working Set 138 139 142 157 141
IO Data KB/sec 0 0 186 2559 0
I am submitting a job using qsub that runs parallelized R. My
intention is to have R programme running on 4 different cores rather than 8 cores. Here are some of my settings in PBS file:
#PBS -l nodes=1:ppn=4
....
time R --no-save < program1.R > program1.log
I am issuing the command ta job_id and I'm seeing that 4 cores are listed. However, the job occupies a large amount of memory(31944900k used vs 32949628k total). If I were to use 8 cores, the jobs got hang due to memory limitation.
top - 21:03:53 up 77 days, 11:54, 0 users, load average: 3.99, 3.75, 3.37
Tasks: 207 total, 5 running, 202 sleeping, 0 stopped, 0 zombie
Cpu(s): 30.4%us, 1.6%sy, 0.0%ni, 66.8%id, 0.0%wa, 0.0%hi, 1.2%si, 0.0%st
Mem: 32949628k total, 31944900k used, 1004728k free, 269812k buffers
Swap: 2097136k total, 8360k used, 2088776k free, 6030856k cached
Here is a snapshot when issuing command ta job_id
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1794 x 25 0 6247m 6.0g 1780 R 99.2 19.1 8:14.37 R
1795 x 25 0 6332m 6.1g 1780 R 99.2 19.4 8:14.37 R
1796 x 25 0 6242m 6.0g 1784 R 99.2 19.1 8:14.37 R
1797 x 25 0 6322m 6.1g 1780 R 99.2 19.4 8:14.33 R
1714 x 18 0 65932 1504 1248 S 0.0 0.0 0:00.00 bash
1761 x 18 0 63840 1244 1052 S 0.0 0.0 0:00.00 20016.hpc
1783 x 18 0 133m 7096 1128 S 0.0 0.0 0:00.00 python
1786 x 18 0 137m 46m 2688 S 0.0 0.1 0:02.06 R
How can I prevent other users from using the other 4 cores? I like to mask somehow that my job is using 8 cores with 4 cores idling.
Could anyone kindly help me out on this? Can this be solved using pbs?
Many Thanks
"How can I prevent other users from using the other 4 cores? I like to mask somehow that my job is using 8 cores with 4 cores idling."
Maybe a simple way around it is to send a 'sleep' job on the other 4? Seems hackish though! (ans warning, my PBS is rusty!)
Why not do the following -
ask PBS for ppn=4, additionally, ask for all the memory on the node, i e
#PBS -l nodes=1:ppn=4 -l mem=31944900k
This might not be possible on your setup.
I am not sure how R is parallelized, but if it is OPENMP you could definitely ask for 8 cores but set OMP_NUM_THREADS to 4
I might be overlooking something due to deadline stress. But this behaviour amazes me.
It looks as if the cursor caches 100 rows and the continue statement flushes the cache
and begins with the first record of a new cache fetch.
I narrowed it down to the following script:
drop table test1;
create table test1 (test1_id NUMBER);
begin
for i in 1..300
loop
insert into test1 values (i);
end loop;
end;
/
declare
cursor c_test1 is
select *
from test1;
begin
for c in c_test1
loop
if mod(c.test1_id,10) = 0
then
dbms_output.put_line(c_test1%ROWCOUNT||' '||c.test1_id||' Continue');
continue;
end if;
dbms_output.put_line(c_test1%ROWCOUNT||' '||c.test1_id||' Process');
end loop;
end;
/
1 1 Process
2 2 Process
3 3 Process
4 4 Process
5 5 Process
6 6 Process
7 7 Process
8 8 Process
9 9 Process
10 10 Continue **Where are tes1_id's 11 to 100?**
11 101 Process
12 102 Process
13 103 Process
14 104 Process
15 105 Process
16 106 Process
17 107 Process
18 108 Process
19 109 Process
20 110 Continue **Where are tes1_id's 111 to 200?**
21 201 Process
22 202 Process
23 203 Process
24 204 Process
25 205 Process
26 206 Process
27 207 Process
28 208 Process
29 209 Process
30 210 Continue **Where are tes1_id's 211 to 300?**
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
PL/SQL Release 11.1.0.7.0 - Production
redhat release 5
2 node RAC
It's a bug: 7306422
Pawel Barut wrote:
http://pbarut.blogspot.com/2009/04/caution-for-loop-and-continue-in-oracle.html
Workaround :
SQL> ALTER SESSION SET PLSQL_OPTIMIZE_LEVEL = 1;
Regards,
Rob