Well the question is clear I guess, We have WebSphere ESB Server 7.5.0.1. We write some java codes to send tasks to scheduler. But what I am wondering is, Is there any scheduler management console to manage and create tasks for schedulers ?
No, there is not. It should be possible to write one yourself using the WASScheduler MBean.
Related
I'm currently using airflow on Amazon Web services using EC2 instances. The big issue is that the average usage of the instances are about 2%...
I'd like to use a scalable architecture and creating instances only for the duration of the job and kill it. I saw on the roadmap that AWS BATCH was suppose to be an executor in 2017 but no new about that.
Do you know if it possible to use AWS BATCH as an executor for all airflow jobs ?
Regards,
Romain.
There is no executor, but an operator is available from version 1.10. After you create an Execution Environment, Job Queue and Job Definition on AWS Batch, you can use the AWSBatchOperator to trigger Jobs.
Here is the source code.
Currently there is a SequentialExecutor, a LocalExecutor, a DaskExecutor, a CeleryExecutor and a MesosExecutor. I heard they're working on AIRFLOW-1899 targeted for 2.0 to introduce a KubernetesExecutor. So, looking at Dask and Celery it doesn't seem they support a mode where their workers are created per task. Mesos might, Kubernetes should, but then you'd have to scale the clusters for the workers accordingly to account for turning off the nodes when un-needed.
We did a little work to get a cloud formation setup where celery workers scale out and in based on metrics from cloud-watch of the average cpu load across the tagged workers.
You would need to create a custom Executor (extended from BaseExecutor) capable of submitting and monitoring the AWS Batch jobs. Also may need to create a custom Docker image for the instances.
I found this repository in my case is working quite well https://github.com/aelzeiny/airflow-aws-executors I'm using Batch jobs with FARGATE_SPOT with computation engine.
I'm just struggling with the logging on AWS CloudWatch and the return status in AWS batch but from Airflow perspective is working
I'd like to develop a simple solution using .NET for the following problem:
We have several computers in a local network:
10 client computers that may need to execute a program that is only installed on two workstations
The two workstations that are only used to execute the defined program
A server that can be used to install a service available from all previously described computers
When a client computer needs to execute the program, he would send a request to the server, and the server would distribute the job to a workstation when available for execution, and inform the client computer when the execution has been performed.
I'm not very used to web and services development so I'm not sure if it's the best way to go, but below is a possible solution I thought about:
A web service on the server stores in queues or in a database the list of tasks with their status
The client computer calls the web service to execute a program and gets a task id. Then calls it every second with the task id to know if the execution has been performed.
The workstations that are available call the web service every second to know if there is something to execute. If yes, the server assigns the task, and the workstation calls the web service when the execution is completed.
I summarized this in the below figure:
Do you think to a simpler solution?
Have a look at signalr! You could use it as messaging framework and you would not need to poll the service from 2 different diretions. With signalR you would be able to push execution orders to the service and the service will notify the client once the execution has been processed. The workstation would be connected with signalR, too. They would not need to ask for execution orders as the webservice would be able to push execution orders to either all or a specific workstation.
I need to pause a SFTP Connector from polling and then restart it after a specified interval, in Mule ESB CE 3.2.1.
Do you know if it's possible, by using Mule's API if necessary?
And, if it was possible, how could I disconnect and reconnect an SFTP client?
Thanks in advance,
Gabriele
I suggest you first try to practice pausing/stopping/starting through JMX, in order to determine exactly what Mule component you want to interact with (the SFTP connector? the flow that uses the connector? just the SFTP inbound endpoint?).
When you have decided exactly what it is you want to do, you can then either do it:
Through JMX calls in the JVM directly on the Mule MBeans,
Through Mule API calls starting by locating the target components in the registry.
I want to add quartz scheduling to an ASP.NET application.
It will be used to send queued up emails.
What are the pros and cons of running quartz.net as windows service vs embedded.
My main concern is how Quartz.NET in embedded mode handles variable number of worker processes in IIS.
Here are some things to you can consider while you decide whether you should run embedded or not:
If you are going to be creating jobs ONLY from within the hosting application, then run embedded. Otherwise, run as a service.
If your jobs might need permissions that are different from the permissions that the web app has, run as a service.
If your jobs are long running jobs, or jobs that use a lot memory, run as a service.
If you need to run your jobs in a clustered environment for either performance, scalability or fault tolerance, run as a service.
From the items above you can deduce that my preference is to run it as a service. This is because if you are going to go through the trouble of setting up a job scheduler, this means that you have jobs that need to run on a schedule, or long running jobs. A service is usually the better choice for this type of work.
Quartz.NET can be instantiated on a per-application basis (web farm configuration mandates number of schedulers). You can safely run multiple schedulers if you have your jobs backed in a database and you have Quartz.NET configured in clustered mode (and clocks synced naturally).
The main concern is to the application pool handling prior to IIS 7.5. Without constant checks your application worker can get recycled and your scheduler will be down until someone issues a web request to fire up the application pool again. IIS 7.5 has the new feature to keep application pools running all the time.
Otherwise there should not be a big difference between the two models.
I normally works in asp.net. But recently I was testing Google App Engine and I found TaskQueues: it's very interesting and powerful. Does anyone know a similar service for asp.net?
I know MSQueue but it's not what I need. I need something like GAE TaskQueue: I put an URL in queue and the URL is triggered (based on queue config).
TyphoonAE is using RabbitMQ to simulate the taskqueue, RabbitMQ provides a .Net client.
http://www.rabbitmq.com
You could try Quartz.NET maybe - http://quartznet.sourceforge.net/
Apache ActiveMQ from version 5.4 has a persistent scheduler built into the message broker.
http://activemq.apache.org/delay-and-schedule-message-delivery.html
ActiveMQ supports a variety of Cross Language Clients and Protocols from Java, C, C++, C#, Ruby, Perl, Python, PHP.
You can set a message to wait with an initial delay, and then repeat delivery 10 times, waiting 10 seconds between each re-delivery
You can also use CRON to schedule a message