I am trying to get a more responsive idea of the system run queue length to see if load balancing based on the one minute load average from sysinfo() is having issues caused by the client processes perhaps looking in lockstep...
I've managed to find /proc/schedstats, and it looks to be what I'm looking for, but...
I want to make sure I base my values on the actual interval between polls of /proc/schedstat, instead of potential processing overhead (it's a shell script).
Now for the question: What is the unit of measurement used for the "timestamp" value at the top of the /proc/schedstats file? It's sure not nanoseconds, because the value is somewhere between 258 and 260 when my script loops through with a sleep 1 between loops.
Inspecting kernel sources sched/stats.c shows that the timestamp field is in jiffies.
What is the unit of measurement used for the "timestamp" value at the top of the /proc/schedstats file?
It's 1/HZ second, typically with HZ=300, the unit would be 3.333 miliseconds if I'm counting right.
Related
I have some COT data that I want to plot under the main price window as an indicator. The COT data are external data, i.e. independent of the prices. So one can not write it like a traditional indicator calculated from the prices. Since I have all the data needed, I don't need to do any calculation. I only need to convert the date and time so that it aligns with the price chart. I will figure out how to do it later. Now, if we ignore the alignment, what I want to ask is how could I plot the data under the price chart? Thanks!
Alternative A:
Use the MT4-GUI tools and plot the data programmatically right into the MT4.Graph or using the screen-layout-plane of GUI-objects, independent of the underlying live-[TimeDOMAIN,PriceDOMAIN]-graphing, both using Expert Advisor-type of MQL4-code. We use this approach most often for all the tasks, that would normally land as a Custom Indicator-type of MQL4-code, as the New-MQL4.56789 code-execution engine has reduced the achievable performance for all ( yes, ALL ) Custom Indicator code-units' execution into a single, thus both RealTime-sensitive and potentially blocking, thread.
Using this alternative, you retain the full freedom of the code-design and may benefit a lot from pre-computing & pre-setting the GUI-objects inside OnInit(){...} section, before entering the trading-loop. This also minimises the latency costs associated with a need to update the GUI-scene from inside an OnTick(){...} event-loop.
Alternative B:
One may also opt to do a similar job using an independent Script-type of MQL4-code unit, as the COT data are weekly announced and thus static per-se.
Launching Script is a step, that can happen whenever feasible and this implementation model may also enjoy some ex-post modification tools, that could be run from another Expert Advisor or another Script MQL4-code, for the sake for some ex-post live-GUI-scene modification/maintenance.
Alternative C:
If one indeed insists to do so, the GUI-composition might be assembled inside a rather special-purpose live-calculated Custom Indicator-type of MQL4-code.
This approach but has to carefully deploy the GUI-composition into the Custom Indicator OnInit(){...} section and avoid any risk of blocking a flow of execution inside the above said critical section of OnCalculate(){...}.
Buffer-mapped, register-based Custom Indicator data & graphing tools are rather rigid for more advanced purposes, that do not strictly follow the hard-wired logic of a code, responding just to a stream of MarketEvent-s, which may, but need not, happen at once, but is being arranged by a sort of mini-batches, so as to process the whole depth of the DataStore in a segmented ( thus less-blocking ) processing approach.
Building the GUI-scene inside the OnInit() section of the Custom Indicator, one may still benefit from distributed processing, if external data source is to be read and/or any similar type inter-platform communications ( be it for a messaging or a signalling purpose ).
My choice would be the [A]
Mapping { Date, Time } onto a MQL4-datetime is trivial, MQL4 used to use since its beginning datetime as int seconds elapsed since 1970-01-01,00:00:00.000 - so simple, so easy.
declare the indicator buffer:
double ExtBufferCOT[];
assign indexes of buffers
SetIndexStyle( 0, DRAW_LINE );
SetIndexBuffer( 0, ExtBufferCOT );
in the OnCalculate() function - make sure it is time to check the levels again ( I think you do not need to update them every tick, right? Maybe once a day or once a week) and then read the file that you have ( we do not have example of file so senseless to describe how to do that here ), convert elements of the file, using StrToTime() and StrToDouble()
the last step - get last N lines from your file, and map them to the indicator buffers:
double value;
datetime time; // - your values from file are here
int shift = iBarShift( _Symbol, 0, time );
ExtBufferCOT[shift] = value; /* probably need to fill buffer
of next candles too
if your chart timeframe
is smaller then frequency
of observations in the file
*/
I have 2 moderate-size datasets that I am using in R. I want to check one dataset if its referenece number matches with the reference numbers in the other dataset and if so, allot a column in the second dataset which contains the value present in the column in the other dataset.
ghi2$state=ifelse(b1$accntnumber %in% ghi2$referencenumber,b1$address,0)
Every time I am running this code, my RStudio hangs up and is unresponsive for a long time. Is it because its taking the time to process the command or is my command wrong.
I am using a 2GB RAM system so I think R hangs up. Should I use the == operator instead of %in%? Would I get the same result?
1. Should I use the == operator instead of %in%?
No (!). See #2.
2. Would I get the same result?
No. The order and position have to match with ==. Also, see #Akrun's comment.
3. How to make it faster and/or deal with RStudio freezing
If RStudio freezes you can save your log file info, send it to the RStudio team who will quickly respond, and also you could bring your log files here for help.
Beyond that, general Big Data rules apply. Here are some tips:
Try data.table
Try it on the command line instead of RStudio
Watch your Resource Monitor (or whatever you use to monitor resources) and observe the memory and CPU usage
If it's a RAM issue you can
a. use a cloud account to get more RAM
b. buy some more RAM (just sayin')
c. use 64-bit R and increase the RAM available to R to its max if it's not already
If it's a CPU issue you can consider parallelization
If any of these ID's are being repeated (and this makes sense in the context of your specific use-case) you can use unique to avoid redundant comparisons
There are lots of other tips you can find in pre-existing Big Data Q&A's on SO as well.
I spent more than two months with RRDTOOL to find out how to store and visualize data on graph. I'm very close now to my goal, but for some reason I don't understand why it is happening that some data are considered to be NaN in my case.
I counting lines in gigabytes sized of log files and have feeding the result to an rrd database to visualize events occurrence. The stepping of the database is 60 seconds, the data is inserted in seconds base whenever it is available, so no guarantee the the next timestamp will be withing the heartbeat or within the stepping. Sometimes no data for minutes.
If have such big distance mostly my data is considered to be NaN.
b1_5D.rrd
1420068436:1
1420069461:1
1420073558:1
1420074583:1
1420076632:1
1420077656:1
1420079707:1
1420080732:1
1420082782:1
1420083807:1
1420086881:1
1420087907:1
1420089959:1
1420090983:1
1420094055:1
1420095080:1
1420097132:1
1420098158:1
1420103284:1
1420104308:1
1420107380:1
1420108403:1
1420117622:1
1420118646:1
1420121717:1
1420122743:1
1420124792:1
1420125815:1
1420131960:1
1420134007:1
1420147326:1
1420148352:1
rrdtool create b1_4A.rrd --start 1420066799 --step 60 DS:Value:GAUGE:120:0:U RRA:AVERAGE:0.5:1:1440 RRA:AVERAGE:0.5:10:1008 RRA:AVERAGE:0.5:30:1440 RRA:AVERAGE:0.5:360:1460
The above gives me an empty graph for the input above.
If I extend the heart beat, than it will fill the time gaps with the same data. I've tried to insert zero values, but that will average out the counts and bring results in mils.
Maybe I taking something wrong regarding RRDTool.
It would be great if someone could explain what I doing wrong.
Thank you.
It sounds as if your data - which is event-based at irregular timings - is not suitable for an RRD structure. RRD prefers to have its data at constant, regular intervals, and will coerce the incoming data to match its requirements.
Your RRD is defined to have a 60s step, and a 120s heartbeat. This means that it expects one sample every 60s, and no further apart than 120s.
Your DS is a gauge, and so the values you enter (all of them '1' in your example) will be the values stored, after any time normalisation.
If you increase the heartbeat, then a value received within this time will be used to make a linear approximation to fill in all samples since the last one. This is why doing so fills the gaps with the same data.
Since your step is 60s, the smallest sample time sidth will be 1 minute.
Since you are always storing '1's, your graph will therefore either show '1' (when the sample was received in the heartbeart window) or Unknown (when the heartbeat expired).
In other words, your graph is showing exactly what you gave it. You data are being coerced into a regular set of numerical values at a 1-minute step, each being 1 or Unknown.
I have a "succeeded" metric that is just the timestamp. I want to see the time between successive successes (this is how long the data is stale for). I have
derivative(Success)
but I also want to know how long between the last success time and the current time. since derivative transforms xs[n] to xs[n+1] - xs[n], the "last" delta doesn't exist. How can I do this? Something like:
derivative(append(Success, now()))
I don't see any graphite functions for appending series, and I don't see any user-defined graphite functions.
The general problem is to be alerted when the data is stale, via graphite monitoring. There may be a better solution than the one I'm thinking about.
identity is a function whose value at any given time is the timestamp of that time.
keepLastValue is a function that takes a series and replicates data points forward over gaps in the data.
So then diffSeries(identity("now"), keepLastValue(Success)) will be a "sawtooth" series that climbs steadily while Success isn't updated, and jumps down to zero (or close to it — there might be some time skew) every time Success has a data point. If you use graphite monitoring to get the current value of that expression and compare it to some threshold, it will probably do what you want.
I need to test my C++ program, which uses system time.
The program is large and it uses third party libraries which possibly also use system time.
I want to see how my program behaves for different dates / times.
Is it possible to change the system time only for one running process in UNIX?
Many thanks...
Well, I found the answer by myself.
In unix shell there is an environmental variable TZ which is the timezone and it is used by all C/C++ time functions.
This variable can be manipulated to set time in current unix shell to any arbitrary time (limited to number of seconds, not milliseconds) and even change date.
Examples:
export TZ=A-02:10:20
(shifts time by 2 hours 10 minutes and 20 seconds from now forward).
export TZ=A+02:10:20
(the same shift backwards)
If you want to change the date, you can use large number of hours, for example:
export TZ=A-72:10:20
Unfortunately it does not let you change date too much, from my experiments it is up to several days back/forward. So for changing month/year this does not work.
(use 'date' command to check current date/time after setting TZ variable).
To cancel all changes use
export TZ=
I guess you can use libfaketime.