I need to programmatically start some autosys jobs. Is there any api available in Java, perl etc for Job management other then usual command line approach.
I m using Autosys r11.
Yes, Autosys R11 has a Java API. See the ca.com web site and download the CA Workload Automation AE API Reference guide for the documentation.
CA Workload Automation AE is CA's latest rename of Autosys
Use a calendar to tell Autosys what day of month you need to run the job.
The commands to launch calendar tool are autocal (graphical editor) and autocal_asc (CLI editor). It should be in $/etc/autosys on ur server.
There is a perl library CA::WAAE which can get job info and/or start jobs. Another library CA::AutoSys is for an older version of Autosys.
Related
Currently some of the jobs are running in different Windows VM's.
for eg.,
Task Scheduler to run
Powershell files
.bat files
python files.
Sql Agent jobs
To run SSIS packages
We are planning to use Airflow to trigger all these jobs to have better visibility and manage dependencies.
Our Airflow in Ubuntu.
I would like know if there is any way to trigger above mentioned jobs in Windows via Airflow.
Can I get some examples on how to achieve my objectives? Please suggest what packages/libraries/plugins/operators I can use.
Yes there is. I would start by looking into the winrm operator and hook that you find in under Microsoft in providers:
http://airflow.apache.org/docs/apache-airflow-providers-microsoft-winrm/stable/index.html
and maybe also:
https://github.com/diyan/pywinrm
May I know if AirFlow support Mainframe jobs ? Can we schedule Mainframe jobs using AirFlow ?
Thanks in advance.
I do not know airflow specifically, but we have used Ansible, Jenkins, and IBM Urban Code Deploy for orchestration that includes distributed and mainframe process parts.
You can SSH into z/OS and use Bash, Python, cURL, Node.js, or Groovy. You could submit JCL via REST APIs. There is a command line processor for Db2 to execute SQL and stored procedures via bash terminal. There is the new Zowe CLI that brings a modern command line interface to z/OS.
I would ask the question - what is the nature of what you want to be scheduled? What language is it written in, or what language do you want it written in? If something exists today, what is the process and how is it scheduled today?
While I haven't used airflow, you can use modern interfaces to do things on z/OS, and frequently that is what is actually needed to integrate with orchestration tools.
Elaborating on Patrick Bossman's good summary, Apache Airflow definitely supports SSH connections to run commands and/or transfer files:
https://airflow.apache.org/howto/connection/ssh.html
z/OS includes OpenSSH as a standard, IBM supported feature in the base operating system at no additional charge, although it's possible it's not running in your particular z/OS installation. Dovetailed Technologies has published a helpful "Quick Install Guide" that explains how to configure and start OpenSSH on z/OS if it isn't configured already:
http://dovetail.com/docs/pt-quick-inst/pt-quick-inst-doc.pdf
Their reference points to IBM's official z/OS documentation if you need more information.
You may decide to have other connections to z/OS from Apache Airflow, but SSH is certainly an available option.
FYI, it appears possible to run Apache Airflow directly on z/OS 2.4 itself. I haven't personally tried it, but it looks good to go. The recipe to do that would be as follows:
Configure and fire up the z/OS Container Extensions ("zCX"), a standard, included, IBM supported, no additional charge feature in z/OS 2.4 that's compatible with IBM z14 and higher model IBM Z machines.
Install and run a Python container (Docker/OCI format) on zCX, for example a Python container from DockerHub. You'll need a Python container image that includes "s390x" architecture support, either on its own or in a multi-architecture container. (No problem with DockerHub's image.)
Use pip to install Apache Airflow within your Python container, per normal.
Configure your SSH (and perhaps other) connection(s) from Airflow to the rest of z/OS, as described above.
You can also run Apache Airflow on Linux on Z/LinuxONE, either on the same IBM Z machine where z/OS runs or on a different machine. You can test Apache Airflow using the free (for up to 120 days) IBM LinuxONE Community Cloud, and you could even create your own custom Docker/OCI container on the LinuxONE Community Cloud for deployment to zCX.
It might even be possible to run Airflow on Python for z/OS, without zCX, although if so there'd be some more work involved. Python for z/OS is available from Rocket Software here:
https://www.rocketsoftware.com/product-categories/mainframe/python-for-zos
From documentation, it isn't very clear whether oozie can schedule and control jobs outside of Hadoop? Can someone shed some light on this? If not, is there any open source based workflow engine which can do that?
Try consider using chronos (from airbnb) advanced version of cron with a UI, built on top of mesos. airbnb.github.com/chronos/
Cheers.
I believe no. Because Oozie itself does not have a resource management policy, all it does is submitting jobs to Hadoop's job tracker at the right time. Besides, for each Oozie workflow, there will be one launcher job which is responsible for submitting the real jobs in the workflow to Hadoop. The launcher job is itself a Hadoop job. So, I think for the versions earlier than Oozie 3.2, the answer should be no.
You might consider trying azkaban by linked in. It was specifically built for hadoop. But unix commands can be specified in the job file of azkaban. So you may develop a workflow for any application(s) that can be run using command line.
I've been working on a new workflow engine called Soop. https://github.com/radixCSgeek/soop it is very lightweight and simple to setup and run using a cron-like syntax. It can run any Java POJO as well as running shell processes, so you can kick off a bash script or whatever.
I am trying to write a script to deploy asp.net application from Clear Case. I am using Clear Case Remote Client.
How will i start? what is the easiest way?
CCRC is for accessing code from a "web" ClearCase snapshot view.
Being a light ClearCase installation, you:
won't have all the cleartool command which would allow to detect new content (new versions on files) to be updated
won't have the easy integration you could have with TeamCity, or Jenkins, or Hudson, ... since they all rely on a cleartool command.
TeamCity, for instance, has still a pending ticket on CCRC support.
For you, since you don't want/need to use those schedulers anyway, you can start by using the CCRC CLI (rcleartool) in order to:
update your ccweb view
check if the update has gotten any new versions
deploy your app if it has gotten anything new.
rcleartool update [-username user-name][-ser/ver server-url][-pas/sword user-password]
[-print] [-ove/rwrite | -nove/rwrite | -ren/ame]
[pname ...]
Jenkins currently follows a similar path to plan for CCRC support: ticket 5192:
(and neither Jenkins nor Hudson support CCRC yet)
I'm thinking about which is better the calling of rcleartool as external tool, or develop a teamapi (or as they call now cmapi) based pure java extension.
More details on this IBM article:
"Continuous integration with IBM Rational ClearCase Remote Client"
In this general architecture schema for CI with CCRC, my suggestion above (rcleartool update) is illustrated by the link between the CM server and the build server.
Personally I'd start by not re-inventing the wheel.
Team City is one such product that can do what you're asking about
http://www.jetbrains.com/teamcity/
Is there a way to stop and start a BizTalk application programmatically?
Ideally something that can be run in a script or batch file and as simple as possible?
I've looked at WMI, but from what I read it isn't able to stop and start Applications? Is this true?
I've also looked at ExplorerOM, but does it work in 64-bit environments?
If your goal is to run it via script or batch file the PowerShell provider for BizTalk should be a good fit for you.