I am unsatisfied with ~ ;task1; task2; so I want to implement my very own task that watches for changes and executes tasks. I.e. I need an sbt task that watches for some files and runs some tasks depending on what is changed.
e.g.
val task1: Initialize[Task[Int]] = ....
val task2: Initialize[Task[Int]] = ....
myTask := {
log.info("Press Enter to stop watching...")
while(isEnterNotPressedYet) {
if (someFilesChanged)
execute(task1) //start task1 and wait for its termination
else if (someOtherFilesChanged)
execute(task2)
Thread.sleep(watchDuration.value)
}
}
task1.value will not work because it will execute task1 BEFORE the body and only once. dynTask will not work because I want to execute tasks multiple times and without leaving the loop. Precisely, question is how to implement following:
def execute[T](task: Initialize[Task[T]]): T
def isEnterNotPressedYet: Boolean
Background:
I have web application that uses JS, Scala, sbt-revolver. Some resources support hot reloading (but still require compilation!), some does not. I.e. if *.js files change, I want to invoke compileJs task. If .scala files change, I want to invoke re-start task. But sbt watch has only one set of watched resources per project...
Had to dive into SBT sources and implement it on my own.
There are quite a bit workarounds around strange SBT behavior but it works!
https://github.com/scf37/sbt-overwatch
Related
Would someone let me know if there is a way to override default failure notification method.
I am planning to send failure notification to SNS, however this means I will have to change all the existing DAG and add on_failure_callback method to it.
I was thinking if there is a way I can override existing notification method such that I don't need to change all the DAG.
or configure global hook for all the dags, such that I don't need to add on_failure_callback to all the dags.
You can use Cluster policy to mutate the task right after the DAG is parsed.
For example, this function could apply a specific queue property when using a specific operator, or enforce a task timeout policy, making sure that no tasks run for more than 48 hours. Here’s an example of what this may look like inside your airflow_local_settings.py:
def policy(task):
if task.__class__.__name__ == 'HivePartitionSensor':
task.queue = "sensor_queue"
if task.timeout > timedelta(hours=48):
task.timeout = timedelta(hours=48)
For Airflow 2.0, this policy should looks:
def task_policy(task):
if task.__class__.__name__ == 'HivePartitionSensor':
task.queue = "sensor_queue"
if task.timeout > timedelta(hours=48):
task.timeout = timedelta(hours=48)
The policy function has been renamed to task_policy.
In a similar way, you can modify other attributes, e.g. on_execute_callback, on_failure_callback, on_success_callback, on_retry_callback.
The airflow_local_settings.py file must be in one of the directories that are in sys.path. The easiest way to take advantage of this is that Airflow adds the directory ~/airflow/config to sys.path at startup, so you you need to create an ~/airfow/config/airflow_local_settings.py file.
I have a class with the delegates for a URLSession. I intend to use it with a background configuration. I understand that the handlers are called when a certain event happens, such as didFinishDownloadingTo.
However, I do have the handle function on my ExtensionDelegate class:
func handle( _ handleBackgroundTasks:
Set<WKRefreshBackgroundTask>)
// Sent when the system needs to launch the application in the background
to process tasks. Tasks arrive in a set, so loop through and process each one.
for task in handleBackgroundTasks {
switch task {
case let urlSessionTask as WKURLSessionRefreshBackgroundTask:
I wonder: where should I handle the data I receive after a download? At the didFinishDownloadingTo or at that function on my ExtensionDelegate class, on the appropriate case of the switch statement?
Another question on the same cycle: I read everywhere that one must remember to setTaskCompleted() after going through the background tasks. But I read elsewhere that one should not set a task as completed if the scheduled data transfer hasn't finished. How do I check that?
There is a very good explanation here.enter link description here
It worked when I had an array with my WKURLSessionRefreshBackgroundTask. Then, at the end of my didFinishDownloadingTo, I get the task on that array that has the same sessionIdentifier as the current session.configuration.identifier, and set it as complete.
I'm trying to find the way to keep the database updated, but the method which does it consumes a lot of time so I try to create a background task to do it.
I searched for solutions and I read this article of different options to run background processes: https://www.hanselman.com/blog/HowToRunBackgroundTasksInASPNET.aspx
But I don't know what's is the best solution out of those, like I'm trying to execute it outside the application. I found something about creating a Windows Service too, but I don't know how, I didn't manage to find some good examples.
What is the best way to keep my database updated everytime I access the application without losing the time it consumes? If you can help me to see the light I would appreciate that so much.
I'm really happy with FluentScheduler, which is handling all my mission-critical scheduling. As well as firing jobs on a scheduled basis, it can also do them on demand, like so:
// Define your job in its own class
public abstract class MyJob : IJob
{
public void Execute()
{
// Do stuff here...
}
}
// Schedule your job at startup
var runAt = DateTime.Today.AddHours(1); // 1am
if (runAt<DateTime.Now)
runAt = runAt.AddDays(1);
Schedule<MyJob>()
.WithName("My Job Name") // Job name, required for manually triggering
.NonReentrant() // Only allow one instance to run at a time
.ToRunOnceAt(runAt) // First execution date/time
.AndEvery(1).Days().At(runAt.Hour, runAt.Minute); // Run every day at the same time
// To manually trigger your job
ScheduledJobRegistry.RunTaskAsync("My Job Name");
I have the scheduled jobs running in a windows Service and use SignalR as a means of triggering them remotely from an MVC Web App when required.
You can use an async method. Just use a void instead of Task.
public async void LongRunningMethod () {
...
// Insert long running code here
...
}
Then call it and it will execute in the background. Be aware that you can have hidden exceptions without proper were handling.
You can also use Hangfire which is a pretty awesome background task scheduler
Here is an example of using Hangfire to run a daily task
RecurringJob.AddOrUpdate(() => Console.Write("Easy!"), Cron.Daily);
I'd like to prevent the following task from getting run multiple times when sbt is running:
val myTask = someSettings map {s => if !s.isDone doSomethingAndSetTheFlag}
So what's expected would be when myTask is run for the first time, isDone is false and something gets done in the task, and then the task sets the flag to true. But when the task is run for the second time, since the isDone flag is true, it skips the actual execution block.
The expected behavior is similar to compile -> when source is compiled, the task doesn't compile the code again the next time it's triggered until watchSource task says the code has been changed.
Is it possible? How?
This is done by sbt, a task will be evaluated only once within a single run. If you want to have a value evaluated once, at the project load time, you can change it to be a SettingKey.
This is documented in the sbt documentation (highlighting is mine):
As mentioned in the introduction, a task is evaluated on demand. Each
time sampleTask is invoked, for example, it will print the sum. If the
username changes between runs, stringTask will take different values
in those separate runs. (Within a run, each task is evaluated at
most once.) In contrast, settings are evaluated once on project load
and are fixed until the next reload.
we need to run some code after the compile step. Making things happen after the compile step seems easy:
compile in Compile <<= (compile in Compile) map{x=>
// post-compile work
doFoo()
x
}
but how do you run something in the freshly compiled code?
More info on the scenario: we are using less for css in a lift project. We wanted lift to compile less into css on the fly (if needed) to help dev, but produce less using the same code, during the build, before tests etc run. less-sbt may help but we are interested in how to solve this generally.
You can use the triggeredBy method like this:
yourTask <<= (fullClasspath in Runtime) map {classpath =>
val loader: ClassLoader = ClasspathUtilities.toLoader(classpath.map(_.data).map(_.getAbsoluteFile))
loader.loadClass("your.class.Here").newInstance()
} triggeredBy(compile in Compile)
This will instantiate your class that has just been compiled, using the runtime classpath for your application, after any compile.
It would probably help if you explained your use scenario for this, since there are some different possible solution paths here and choosing between them might involve considerations that you haven't told us.
You won't be able to just write down an ordinary method call into the compiled code. That would be impossible since at the time your build definition is compiled, sbt hasn't looked at your project code yet.
Warning: rambling and thinking out loud ahead.
One trick I can suggest is to access testLoader in Test to get a classloader in which your compiled classes are loaded, and then use reflection to call methods there. For example, in my own build I have:
val netlogoVersion = taskKey[String]("...")
netlogoVersion := {
(testLoader in Test).value
.loadClass("org.nlogo.api.Version")
.getMethod("version")
.invoke(null).asInstanceOf[String]
}
I'm not sure whether accessing testLoader in Test will actually work in your case because testLoader loads your test classes as well as your regular classes, so you might get a circular dependency between compile in Compile and compile in Test.
If you want to try to make a classloader that just has your regular classes loaded, well, hmm. You could look in the sbt source code at the implementation of createTestLoader and use it for inspiration, modifying the arguments that are passed to ClasspathUtilities.makeLoader. (You might also look at the similar code in Run.run0. It calls makeLoader as part of the implementation of the run task.)
A different path you might consider is to reuse the machinery behind the run task to run your code. You won't be able to call an arbitrary method in your compiled code this way, only a main method, but perhaps you can live with that, if you don't need a return value back.
The fullRunTask method exists for creating entire run-like tasks. See "How can I create a custom run task, in addition to run?" from http://www.scala-sbt.org/0.13.1/docs/faq.html . fullRunTask makes it very easy to create a separate task that runs something in your compiled code, but by itself it won't get you all the way to a solution because you need a way of attaching that task to the existing compile in Compile task. If you go this route, I'd suggest asking it that last piece as a separate question.
Consider bypassing fullRunTask and just assembling your own call to Run.run. They use the same machinery. In my own build, I currently use fullRunTask, but back before fullRunTask was added by sbt, here was what my equivalent Run.run-based code looked like:
(..., fullClasspath in Compile, runner, streams, ...) map {
(..., cp, runner, s, ...) =>
Run.run("name.of.my.MainClass",
cp.map(_.data), Seq(), s.log)(runner)
}
Pardon the sbt 0.12, pre-macro syntax; this would look nicer if redone with the 0.13 macros.
Anyway, hopefully something in this brain dump proves useful.