In order to help me understand, I'm looking for examples of command chaining in Parsley. What I mean by command chaining is when one command returns a result that is then immediately used to initiate another command. I ask this question here because the parsley forums have been down for 2 days.
Have you had a look at the Spicelib Task Framework? It should do everything you want to, and it has some good example.
Related
I want to do hive query using HiveOperator and the output of that query should transfer to python script using PythonOperator. Is it possible and how?
One approach to this kind of problem in general is to use Airflow's xcoms - see documentation
However, I would use this sparingly and only where strictly necessary. Otherwise, you risk ending up with your operators being quite tangled and interdependent.
In Ruby, specifically RSpec, you can tell the test runner to abort on the first test that does not pass by the command-line flag --fail-fast. This helps a lot to not waste time or lose focus when fixing a lot of test in a row, for example when doing test-driven or behavior-driven development.
Now on Elixir with ExUnit I am looking for a way to do exactly that. Is there a way to do this?
There is such an option since Elixir 1.8.
Use the --max-failures switch to limit the number of tests evaluated with failure. To halt the test suite after the first failure, run this:
mix test --max-failures 1
Unfortunately there is (to my knowledge) no such flag implemented.
However, you can run a single test by
mix test path/to/testfile.exs:12
where 12 is the line number of the test.
Hope that helps!
That makes not much sense since tests in Elixir are a) to be run blazingly fast and b) in most cases are to be run asynchronously. Immediate termination of the test suite on the failed test is an anti-pattern and that’s why it’s not allowed by ExUnit authors.
One still has an option to shoot their own leg: just implement a custom handler for the EventManager and kill the whole application on “test failed” event.
For BDD, one preferably uses tags, running the test suite with only this feature included. That way you’ll get an ability to run tests per feature at any time in the future.
Also, as a last resort one might run a specific case only by passing the file name to mix test and/or a specific test only by passing the file name followed by a colon and a line number.
Some libraries, e.g. xml-rpc, directly use url-retrieve. I want them to instead use request.el, so that I can choose curl as my backend. Is there an easy shim-layer I can install?
I'm looking for something like curl-for-url, which transparently rebinds url-http with a compatible implementation. (curl-for-url itself doesn't actually work very well, though.)
You could do this using advice, but you will need to use the
ad-get-arg/ad-get-args functions to extract the arguments url-retrieve was
called with and determine how you want to process them and pass them to the
retrieve function. The one which will likely be problematic is the callback
function. However, provided you can setup the buffer with the downloaded
data in the same way, with the same name as url-retrieve, you should be able to
apply the callback manually after the call to request and you have setup the
buffers as necessary.
It will be a fair bit of work and you will need to dig deep into both the url.el
and request.el libraries. It is also likely to be a bit fragile.
One concern I would have is the use of monkey patching by request.el. From the
project page, it looks like this code has not been updated since Emacs version
25.1 and the current official emacs is 25.2. This is one of the problems with
monkey patching - you need to keep versions in sync to avoid version
incompatibility issues.
It also seems odd to me to have someone who has patches to fix known bugs if
those patches have not been applied to the mainstream version - especially when
there has been a more recent release of the mainstream version.
The first thing I would do is upgrade to emacs 25.2 and then determine if using
request.el is as justified. I would also verify the problems you experience are
actually due to url-retrieve or are perhaps due to callbacks being passed to
that function. If it is a problem with the callbacks, you may be better off
using advice to fix those callbacks rather than replace the underlying
problems.
If you only have issues in some situations where url-retrieve is used, it may
also be easier to go up one level and look at the things which are using it and
perhaps use something like advice to replace the call to url-retrieve with
request at that level.
Someone might be able to provide more specific recommendations if you provide
more detail on the precise reasons you cannot or do not want to use the
url.el library.
I am making an app, that lets you play music synchronously on different systems. For the project, I have decided to use twisted, PyGtk2, Pyglet. I am confused on how should the main loop be run. Should i run pyglet's loop in a separate thread or should i implement a new reactor integrating twisted, pygtk2, pyglet. Will the performance suffer if i try to integrate three loops together?
I used https://github.com/padraigkitterick/pyglet-twisted when playing with pyglet and twisted, and it worked for my toy cases. Good starting point, anyway.
The above is a new reactor based on ThreadedSelectReactor.
It's not clear to me what the composition of all three would look like...
Twisted already has a solution for integrating with gtk:
http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html#core-howto-choosing-reactor-gtk
I'm not familiar with pyglet but if it has a main loop like GTK then both of your ideas seem feasible. You could also look into how twisted implements the GTK integration explained in the link above and try to replicate that for pyglet.
I'm very new to unix programming, so please bear with me. :)
I'd like to pass data between two processes. I was going to use named pipes, but read about these "half-duplex" pipes, and it was very intriguing, so I figured that I would give them a try first.
I have two issues with these pipes thus far:
I haven't figured out how to get execlp to run another application from my child process
Even if I could, debugging is tough because I've only been able to set breakpoints in the parent process
I'm sure there are reasons for these issues. I am starting to wonder if it makes sense to just forget about them and just use named pipes so I can debug each application in a separate instance of eclipse.
If there is any relevant information, please let me know. The code I am using is essentially what it found on tldp.org.
EDIT -- I renamed my question to be about unix pipes in general. I had assumed that for named pipes, I wouldn't have to use fork(), but all of the examples I have seen so far require it. So regardless of half-duplex or named pipes, I'm going to need to be able to debug the child process somehow!
EDIT #2 -- this example clearly shows that what I had seen before (on an IBM link) regarding named pipes wasn't necessarily true.
I recommend two tools:
strace -ff should give you a trace of all significant events, allowing you to examine in detail what's going on, namely, all reads and writes;
lsof allows you to dump file descriptors of involved processes, clearly showing what is connected to what else and, in particular, if you forgot to close() some descriptors and the whole thing deadlocks.