Due to a few company-specific features, which I need to swap in and out, I sometimes have migrated scripts which are not present in the sql directory when I run "info" or "migrate" at a later time. I just noticed an inconsistency, though, in how this displays:
+----------------+----------------------------+---------------------+---------+
| Version | Description | Installed on | State |
+----------------+----------------------------+---------------------+---------+
...
| 4.1 | Add new reports synonyms | 2013-05-31 16:38:22 | Success |
| 4.1.1 | BRNC Add new reports synon | 2013-05-31 16:38:22 | Missing |
| 4.2 | Convert old DATA to DATA2 | 2013-05-31 16:38:22 | Success |
| 4.2.1 | BRNC Convert old DATA to D | 2013-05-31 16:38:22 | Future |
+----------------+----------------------------+---------------------+---------+
So, "Success" means that scripts have been run, and "Missing" means they were run and are no longer present. But what does "Future" mean?
This is similar but not identical to a question:
state of migration scripts is "future"
which was never officially answered, but where Axel Fontaine said in a comment that this had been fixed. I checked, and my jars (3/18) are a later date than his comment (3/2).
As it currently stands this is what these mean:
missing -> executed, no longer found in configured locations, older than the newest found script
future -> executed, no longer found in configured locations, newer than the newest found script
Coming to think of it though, I feel this minor distinction might not be worth a separate state in the info results. I will revisit this in time for 2.2.
Related
Background: I'm confuse with Erlang's scheduler for a long time until have a look at The Beam Book. I had do some research on async/non-blocking programming in some language(Elixir/Erlang, Scala/Java, Golang) which include Actor pattern, Future/Promise, coroutine mainly. Coroutine and Actor pattern is similarly in term of they can both think as lightweight process.
The Problem
I'm find a weakness of the async programming: It will cause a lightweight-process(or a task) re-queue to the end of the scheduler Ready Queue if it invoke any of the block operation. The block operation will not block OS-Thread but it occurred mainly because of invoke a async action such as 'aio_read'.
re-queue means the process will be put at the end of Scheduler even if the process is just scheduled.In server-side programming, it will make a client request delay with a relatively long time compare with it should process time.The Beam Book give a detail description:
A processs trying to do a receive on an empty mailbox or on a mailbox with no matching messages will yield and go into the waiting state.
When a message is delivered to an inbox the sending process will check whether the receiver is sleeping in the waiting state, and in that case it will wake the process, change its state to runable, and put it at the end of the appropriate ready queue.
The affect could be see in many benchmark test: More request, more response time for every request.
A good scheduler should make the response time nearly the real process time of the request if ignore OS-Thread Context Switch.
I haven't seem others discuss the aspect yet.
As a conclusion, there are two question:
1. I want make a confirm whether re-queue problem really exist in async-programming world.
2. Besides, Is Erlang really has the problem if it handle tens of thousands of the process, especially use many GenServer.call in a request-response-chain?
Your answers are here: https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html
Reposting it here, allowing edits.
Erlang Scheduling
Erlang as a real-time platform for multitasking uses Preemptive Scheduling. The responsibility of an Erlang scheduler is selecting a Process and executing their code. It also does Garbage Collection and Memory Management. The factor of selecting a process for execution is based on their priority level which is configurable per process and in each priority level processes are scheduled in a round robin fashion. On the other hand the factor of preempting a process from execution is based on a certain number of Reductions since the last time it was selected for execution, regardless of its priority level. The reduction is a counter per process that is normally incremented by one for each function call. It is used for preempting processes and context switching them when the counter of a process reaches the maximum number of reductions. For example in Erlang/OTP R12B this maximum number was 2000 reductions.
The scheduling of tasks in Erlang has a long history. It has been changing over the time. These changes were affected by the changes in SMP (Symmetric Multi-Processing) feature of Erlang.
Scheduling Before R11B
Before R11B Erlang did not have SMP support, so just one scheduler was run in the main OS process’s thread and accordingly just one Run Queue existed. The scheduler picked runnable Erlang processes and IO tasks from the run queue and executed them.
Erlang VM
+--------------------------------------------------------+
| |
| +-----------------+ +-----------------+ |
| | | | | |
| | Scheduler +--------------> Task # 1 | |
| | | | | |
| +-----------------+ | Task # 2 | |
| | | |
| | Task # 3 | |
| | | |
| | Task # 4 | |
| | | |
| | Task # N | |
| | | |
| +-----------------+ |
| | | |
| | Run Queue | |
| | | |
| +-----------------+ |
| |
+--------------------------------------------------------+
This way there was no need to lock data structures but the written application couldn’t take advantage of parallelism.
Scheduling In R11B and R12B
SMP support was added to Erlang VM so it could have 1 to 1024 schedulers each was run in one OS process’s thread. However, in this version schedulers could pick runnable tasks from just one common run queue.
Erlang VM
+--------------------------------------------------------+
| |
| +-----------------+ +-----------------+ |
| | | | | |
| | Scheduler # 1 +--------------> Task # 1 | |
| | | +---------> | |
| +-----------------+ | +----> Task # 2 | |
| | | | | |
| +-----------------+ | | | Task # 3 | |
| | | | | | | |
| | Scheduler # 2 +----+ | | Task # 4 | |
| | | | | | |
| +-----------------+ | | Task # N | |
| | | | |
| +-----------------+ | +-----------------+ |
| | | | | | |
| | Scheduler # N +---------+ | Run Queue | |
| | | | | |
| +-----------------+ +-----------------+ |
| |
+--------------------------------------------------------+
Because of the resulting parallelism of this method, all shared data structures are protected with locks. For example the run queue itself is a shared data structure which must be protected. Although the lock can provide performance penalty, the performance improvements which was achieved in multi-core processors systems was interesting.
Some known bottlenecks in this version was as follows:
The common run queue becomes a bottleneck when the number of schedulers increases.
Increasing the involved lock of ETS tables which also affects Mnesia.
Increasing the lock conflicts when many processes are sending messages to the same process.
A process waiting to get a lock can block its scheduler.
However, separating run queues per scheduler was picked to solve these bottleneck issues in next versions.
Scheduling After R13B
In this version each scheduler has its own run queue. It decreases the number of lock conflicts in systems with many schedulers on many cores and also improves the overall performance.
Erlang VM
+--------------------------------------------------------+
| |
| +-----------------+-----------------+ |
| | | | |
| | Scheduler # 1 | Run Queue # 1 <--+ |
| | | | | |
| +-----------------+-----------------+ | |
| | |
| +-----------------+-----------------+ | |
| | | | | |
| | Scheduler # 2 | Run Queue # 2 <----> Migration |
| | | | | Logic |
| +-----------------+-----------------+ | |
| | |
| +-----------------+-----------------+ | |
| | | | | |
| | Scheduler # N | Run Queue # N <--+ |
| | | | |
| +-----------------+-----------------+ |
| |
+--------------------------------------------------------+
This way the locking conflicts when accessing the run queue is solved but introduces some new concerns:
How fair is the process of dividing tasks among run queues?
What if one scheduler gets overloaded with tasks while others are idle?
Based on what order a scheduler can steal tasks from an overloaded scheduler?
What if we started many schedulers but there all so few tasks to do?
These concerns lead the Erlang team to introduce a concept for making scheduling fair and efficient, the Migration Logic. It tries to control and balance run queues based on the statistics that collects from the system.
However we should not depend on the scheduling to remain exactly as it is today, because it is likely to be changed in future releases in order to get better.
As a conclusion, there are two question:
1. I want make a confirm whether re-queue problem really exist in async-programming world.
2. Besides, Is Erlang really has the problem if it handle tens of thousands of the process, especially use many GenServer.call in a
request-response-chain?
Depending which Erlang version are you talking about there are different tradeoffs. Since there are multiple queues in newer Erlang versions your problem does not exist in the form you are specifying it.
I have seen Erlang (and its VM) handle millions of Erlang processes just fine, also used an Erlang based system to handle 100.000+ clients with really tight SLA on the latency. Not sure about the details of your problem, but a reasonably written Erlang/Elixir service can handle the workload you specified unless there is something you left out.
I am getting data from the server in a file (in format1) everyday ,however i am getting the data for last one week.
I have to archive the data for 1.5 months exactly,because this data is being picked to make some graphical representation.
I have tried to merge the the files of 2 days and sort them uniquely (code1) ,however it didn't work because everyday name of raw file is changing.However Time-stamp is unique in this file,but I am not sure how to sort the unique data on base of a specific column also,is there any way to delete the data older than 1.5 months.
For Deletion ,The logic i thought is deleting by fetching today's date - least date of that file but again unable to fetch least date.
Format1
r01/WAS2/oss_change0_5.log:2016-03-21T11:13:36.354+0000 | (307,868,305) | OSS_CHANGE |
com.nokia.oss.configurator.rac.provisioningservices.util.Log.logAuditSuccessWithResources | RACPRS RNC 6.0 or
newer_Direct_Activation: LOCKING SUCCEEDED audit[ | Source='Server' | User identity='vpaineni' | Operation
identifier='CMNetworkMOWriterLocking' | Success code='T' | Cause code='N/A' | Identifier='SUCCESS' | Target element='PLMN-
PLMN/RNC-199/WBTS-720' | Client address='10.7.80.21' | Source session identifier='' | Target session identifier='' |
Category code='' | Effect code='' | Network Transaction identifier='' | Source user identity='' | Target user identity='' |
Timestamp='1458558816354']
Code1
cat file1 file2 |sort -u > file3
Data on Day2 ,the input file name Differ
r01/WAS2/oss_change0_11.log:2016-03-21T11:13:36.354+0000 | (307,868,305) | OSS_CHANGE |
com.nokia.oss.configurator.rac.provisioningservices.util.Log.logAuditSuccessWithResources | RACPRS RNC 6.0 or
newer_Direct_Activation: LOCKING SUCCEEDED audit[ | Source='Server' | User identity='vpaineni' | Operation
identifier='CMNetworkMOWriterLocking' | Success code='T' | Cause code='N/A' | Identifier='SUCCESS' | Target element='PLMN-
PLMN/RNC-199/WBTS-720' | Client address='10.7.80.21' | Source session identifier='' | Target session identifier='' |
Category code='' | Effect code='' | Network Transaction identifier='' | Source user identity='' | Target user identity='' |
Timestamp='1458558816354']
I have written almost similar kind of code a week back.
Awk is a good Tool ,if you want to do any operation column wise.
Also , Sort Unique will not work as file name is changing
Both unique rows and least date can be find using awk.
1 To Get Unique file content
cat file1 file2 |awk -F "\|" '!repeat[$21]++' > file3;
Here -F specifies your field separator
Repeat is taking 21st field that is time stamp
and will only print 1st occurrence of that time ,rest ignored
So,finally unique content of file1 and file2 will be available in file3
2 To Get least Date and find difference between 2 dates
Least_Date=`awk -F: '{print substr($2,1,10)}' RMCR10.log|sort|head -1`;
Today_Date=`date +%F` ;
Diff=`echo "( \`date -d $Today_Date +%s\` - \`date -d $Start_Date +%s\`) / (24*3600)" | bc -l`;
Diff1=${Diff/.*};
if [ "$Diff1" -ge "90" ]
then
Here we have used {:} as field separator, and finally substring to get exact date field then sorting and finding least
value.
Subtracting today's Date by using Binary calculator and then removing decimals.
Hope it helps .....
Due to a few company-specific features, which I need to swap in and out, I sometimes have migrated scripts which are not present in the sql directory when I run "info" or "migrate" at a later time. I just noticed an inconsistency, though, in how this displays:
+----------------+----------------------------+---------------------+---------+
| Version | Description | Installed on | State |
+----------------+----------------------------+---------------------+---------+
...
| 4.1 | Add new reports synonyms | 2013-05-31 16:38:22 | Success |
| 4.1.1 | BRNC Add new reports synon | 2013-05-31 16:38:22 | Missing |
| 4.2 | Convert old DATA to DATA2 | 2013-05-31 16:38:22 | Success |
| 4.2.1 | BRNC Convert old DATA to D | 2013-05-31 16:38:22 | Future |
+----------------+----------------------------+---------------------+---------+
So, "Success" means that scripts have been run, and "Missing" means they were run and are no longer present. But what does "Future" mean?
This is similar but not identical to a question:
state of migration scripts is "future"
which was never officially answered, but where Axel Fontaine said in a comment that this had been fixed. I checked, and my jars (3/18) are a later date than his comment (3/2).
As it currently stands this is what these mean:
missing -> executed, no longer found in configured locations, older than the newest found script
future -> executed, no longer found in configured locations, newer than the newest found script
Coming to think of it though, I feel this minor distinction might not be worth a separate state in the info results. I will revisit this in time for 2.2.
The below is rewarded with a complaint that Remove Directory requires 1 or 2 arguments and I gave it none. I'm using 2.6.3, and dcsLshLocation is a variable (and adding an x in front doesn't change the error). I'm using the Java version of all this.
*** Settings ***
| Documentation | http://jira.basistech.net:8080/browse/JEST-226
| Resource | src/main/resources/jug-shared-keywords.txt
| Force Tags | integration |
| Suite Precondition | Run Keywords |
| | ... | Validate SUT Installations |
| | ... | Launch Derby Server |
| | ... | Copy file ${jddInstallDir}/conf/jdd-conf-basic.xml to ${jddInstallDir}/conf/jdd-conf.xml
| | ... | Remove Directory | ${dcsLshLocation} |
| Suite Teardown | Run Keywords | Shutdown Derby
| Test Timeout | 20 minutes
When this question was originally written, Run Keywords could only run keywords that do not take arguments. That is no longer true. From the documentation:
Starting from Robot Framework 2.7.6, keywords can also be run with arguments using upper case AND as a separator between keywords. The keywords are executed so that the first argument is the first keyword and proceeding arguments until the first AND are arguments to it. First argument after the first AND is the second keyword and proceeding arguments until the next AND are its arguments. And so on.
The code in the question can thus be expressed like this:
| Suite Precondition | Run Keywords |
| | ... | Validate SUT Installations
| | ... | AND | Launch Derby Server
| | ... | AND | Copy file ${jddInstallDir}/conf/jdd-conf-basic.xml to ${jddInstallDir}/conf/jdd-conf.xml
| | ... | AND | Remove Directory | ${dcsLshLocation}
The following is the original answer to the question, which others may still find useful. It is still relevant for versions of robot framework prior to 2.7.6.
When you use Run Keywords, you cannot run keywords that take arguments. Admittedly the documentation is a bit unclear, but this is what it says:
User keywords must nevertheless be used if the executed keywords need
to take arguments.
What it should say is that, when you use Run Keywords, each argument is the name of a keyword to run. This keyword cannot take any arguments itself because robot can't know where the arguments for one keyword ends and the next keyword begins.
Remember that ... simply means that the previous row is continued on the next, so while it looks like each row is a separate keyword with arguments, it's not. You example is the same as:
| Suite Precondition | Run Keywords |
| | ... | Validate SUT Installations |
| | ... | Launch Derby Server |
| | ... | Copy file ${jddInstallDir}/conf/jdd-conf-basic.xml to ${jddInstallDir}/conf/jdd-conf.xml
| | ... | Remove Directory |
| | ... | ${dcsLshLocation} |
From the get go: sorry if I'm not using the proper emacs terminology -- I'm relatively wet behind the ears in the emacs world.
Most of my work in emacs is for programming R, and I'm using ESS and ECB to do so quite happily. I'd like to build a custom ECB layout which uses the entire bottom of the screen as my R console, while putting some ECB-specific buffers on the left.
Using ECB-esque layout diagrams, I'd like my layout to look like pretty much exactly like "left13", except I'd like the entirety of the "compilation" buffer to be my running R console (or any shell, for that matter):
-------------------------------------------------------
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| Directories | Edit |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
-------------------------------------------------------
| |
| R Console |
| |
-------------------------------------------------------
If I can just split my buffer in two (vertically), then call ecb-activate from the top buffer (and not allow it to touch my bottom buffer), I'm imagining it could work (hence the subject of my question).
That doesn't work, though, and I don't know how to get an entier "bottom pane" out of a layout to work in the way I like using trying to use ECB's customize layout functionality.
Does anybody know if/how I can do this?
Short answer: No.
Longer answer: Unfortunately, ECB completely takes over Emacs "window" management at a very low level. So it's all or nothing. You can't comfortably combine it with regular window splitting. What you might be able to do, is to adjust the layout ECB gives you or to program a custom layout. (Some assembly required.)