I am looking for JCL Script/Procedures in mainframe which can facilitate file transfer from Unix server to Mainframe.I am required to do FTPS for the Outbound Jobs (pull the file from UNIX server to mainframe Host).
Rather than a JCL, just do it a shell script. Here is a good site on using such commands:
https://blog.eduonix.com/shell-scripting/how-to-automate-ftp-transfers-in-linux-shell-scripting/
Once you have that working in the shell script in USS, you should be able to call the shell script from a JCL so you can execute it on a scheduled batch job if you need it.
Kenny's suggestion is fairly reasonable. IBM's documentation on how to write JCL for FTP(S)-related tasks is available in their "z/OS Communications Server: IP User's Guide and Commands" publication, IBM Publication No. SC27-3662. The current revision appears to be SC27-3662-30, but later revisions are possible. You can easily find this publication online, and make sure you don't skip the section beginning with the title "Submitting FTP requests in batch." Make sure you set the security options correctly (of course).
Please note that you're asking about FTPS, i.e. TLS encryption applied to either or both (preferably both) of the FTP channels (control and data). SFTP is another file transfer protocol based on SSH that z/OS also supports.
Another possible approach that you'll fairly often find available on z/OS installations is to use IBM MQ Advanced for z/OS's Managed File Transfer (MFT) feature to retrieve the file(s) using FTPS. As the name suggests, this'll be managed and have at least some error handling capabilities.
Yet another possible approach if you prefer HTTPS protocol is to use the z/OS Client Web Enablement Toolkit's HTTPS protocol enabler to fetch the file. That's a built-in, standard feature in all currently supported z/OS releases, and you can use it from a relatively simple REXX script for example. Details are available here (z/OS 2.3 variant of the documentation):
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.ieac100/ieac1-cwe-http.htm
Related
I get a valid connection with phpseclib but because of the server's requirements I must issue a change remote directory command, $sftp->chdir($dir="//ARTDONE.G9876TT1"), to this directory, (exact format, not the actual name). This change directory command works with Putty's psftp.exe as "CD //ARTDONE.G9876TT1" in windows and with WinSCP's "go to this folder GUI input" but not with PHPSECLIB's sftp object method. Is there something about this directory format that needs to change when using phpseclib? The error message is "permission denied", but I get that same message for any other navigation commands.
Is there a way to issue literal sftp commands with phpseclib sftp?
Or can I use $ssh->exec("CD //ARTDONE.G9876TT1") in some way within the $sftp object that I cannot currently imagine?
The phpseclib appends / to the path in SFTP::chdir call. I believe this is what your server does not like.
Note that SFTP does not even use the concept of a working directory. It's faked locally by phpseclib (and other clients like WinSCP or OpenSSH). So you do not really need to use SFTP::chdir. You might instead use absolute paths in all phpseclib API calls. Alternatively, just setting SFTP::pwd has the same effect as calling SFTP::chdir, except that you will bypass the validation that causes you the troubles.
Accessing z/OS Data Sets via SFTP/FTP
Appending a / surely breaks the access. The OP is accessing an IBM z/OS system running an SFTP server.
IBM z/OS
z/OS is kind a hybrid operating system having a traditional MVS based "kernel" (not really named "kernel" in the doc), and a XPG 4.2 compliant UNIX kernel running in parallel. The UINX side supports file systems with directories and files. The MVS side has a completely different "file system", based on data sets which are named in a non-hierarchal system.
The UNIX file system on z/OS
There is not much to say about the UNIX file system on z/OS. Is it XPG compliant, thus the usage is not different to any other UNIX lik system.
The MVS Data Sets on z/OS
As said above, there is the traditional MVS Data Set based "file system" on z/OS, which is quite different to much you know about files and directories on UNIX system.
Disk Space on z/OS is assigned to MVS data sets. Data sets are named using dot separates names, that can be up to 44 characters long. The parts between two dots can be up to 8 characters long.
Examples:
ARTDONE.G9876TT1
ARTDONE.NEXT.DATA.SET
ARTDONE.NEXT.ANOTHER.ONE
SYS1.LINKLIB
ZUSER.SOURCE.REXX
What seems to be a hierarchy in the first three examples, is not. They are unrelated from the physical point of view, though related in a logical.
Note: Slashed / are not valid in MVS data set names.
SFTP/FTP servers on z/OS
SFTP/FPT servers in z/OS mimic the client side view of directories and files when accessing MVS Data Sets in that the dots in the names are kind of treated like slashes in UNIX. I.e. they support pwd and cd based on the dots.
Example:
cd //ARTDONE.NEXT sets the current working directory to ARTDONE.NEXT. A ls the lists all data sets, of which the name starts with ARTDONE.NEXT, i.e.
ARTDONE.NEXT.DATA.SET
ARTDONE.NEXT.ANOTHER.ONE
but not ARTDONE.G9876TT1.
But how would the server know whether a client side "directory" access is meant to access the UNIX or the MVS data world? The // at the beginning of the parameter passed to the server indicates the server shall switch to the MVS data set world.
Conclusion
client side software should be careful when verifying paths that might be sent to z/OS servers. Accept // as a special indicator. Do not append / in all cases.
More Detailes
There is much more behind this topic than can be described here. Read IBM documentation on z/OS if interested. I would recommend Introduction to the New Mainframe: z/OS Basics as a starter.
We have a IBM Host System Z sitting in our cellar. Now the issue is that i have no clue about Mainframes!!! (It's not USS btw.)
The Problem: How can i transfer a file from the host system to a windows machine.
Usually on UNIX systems i would just install and ssh daemon and connect to it via. a program called winscp. After that transfer the file in binary so that it does not convert something (Ultraedit and other Editors can handle this).
With the host system it seems to be a bit difficult as the original format from IBM is EBCDIC and i have no idea if there is a state of the art SFTP server program for the host. Could anybody be so kind and enlighten me? From my current expirience with IT there must be a state of the art sftp connection to that system? I appreciate any help/hints/solutions.
Thank you,
O.S
If the mainframe "sitting in [your] cellar" is running z/OS then it has Unix System Services installed. You can't have z/OS without it.
There is an SFTP package available (for free) for z/OS.
You can test to see about Unix System Services by firing up a 3270 emulator going to ISPF option 3.17, putting a forward slash (/) in the Pathname field and pressing the mainframe Enter key. Another way would be to key OMVS at a TSO READY prompt, which will start up a 3270-based Unix shell.
It is possible that USS is simply not available to you; if you're running any supported release of z/OS then USS is present. There could be concerns about supporting something outside a particular group,
Or, depending on what OS you have running on your System z, it's possible you don't have z/OS. You could have z/VM, you could have zLinux, you could have TPF. However, if you're running zLinux, you have linux, which has sftp installed, and which uses ASCII, not EBCDIC.
As cschneid says, however, if you have z/OS, you have USS. TCP/IP, among other things, won't run without it. Also note that z/OS TCP/IP has an FTP server, so you can connect that way if the FTP server is set up. If security is an issue, FTPS is supported, although it's painful to set up. With the native FTP server, you can convert from EBCDIC to ASCII when you're doing the transfer. There's also an NFS server available. And SMB as well, I believe.
And there's an FTP client available as well, so you could FTP from z/OS to your system, if you wanted to.
Maybe a better thing to do would explain what you're trying to do with the data, and what the data is, in general. You can edit files directly on the mainframe, using either TSO, ISPF, or OMVS editors. There are a lot of data types that the mainframe supports that you're not going to be able to handle on a non-z system unless you go through an export process. I'm not really clear on whether you want to convert the file to ASCII when you transfer it or not.
While the others are correct that all recent releases of z/OS have USS built-in, there's quite a bit of setup work that needs to be done in order for individual users to have access to USS capabilities like SFTP. Out of the box, you get USS "minimal mode" that just has enough of USS to support the TCP/IP stack and so forth. USS "full function mode" requires setup:
HFS filesystems need to be allocated
Your security package needs to be manage UIDs/GIDs for your users
etc etc etc
Still, with these details and with nothing more than the software you're entitled to as part of your z/OS license, you can certainly run SFTP and all the other UNIX style network services you're used to.
A good place to start is the UNIX Services Planning guide: http://publibz.boulder.ibm.com/epubs/pdf/bpxzb2c0.pdf
So, I've taken a handful of programming courses(object-oriented, web) but never had "hands-on" projects where it's outside of coding.
Now I'm trying to figure out what these SSH stuff is about, I can't even figure out which client to use, so picked filezilla for now.
My question is, where can I read more about these terms like ports, and whatnots, in a way so I'm not learning aimlessly.
Thanks!
Basically, SSH is a way to command another computer exactly what to do over the Internet. You can execute any commend the remote system has, and your user has permission for.
The Internet
The Internet runs on a series of protocols collectively named TCP/IP. TCP/IP defines a way to find and address individual computers (IP) and a way to communicate between them (TCP).
You can think of computers on the Internet as a large collection of office buildings all close together. Each office has the exact same number of windows: 65535. Offices (computers) communicate by stringing channels between windows (ports). Each channel has two ends, called sockets. Each socket is associated with a port on the respective computer. We send data back and forth, and then the connection is closed.
Client/Server
There are two types of computers on the Internet: clients, and servers. Clients request information, and servers provide it. Ports 1-1024 are reserved for servers, 1 port per protocol. The full list is here, and as you can see, it is not without contention.
Let's say you visit a website
Your browser, the client program, sees that you typed "stackoverflow.com", and using DNS, discovers that stackoverflow.com is computer number 64.34.119.12. This is it's IP address. It allows your computer to find the network stackoverflow.com is located in, route to it, and establish a connection to the Stack Overflow web server. The web server is a program that accepts client requests from a browser like yours.
They speak in a protocol called HTTP - it allows your browser to request a page determined by a URL. The server sees the request, runs a program to construct a web page (or retrieves an HTML file, image, or any other file), and sends the result back to the browser. Port 80 has been reserved for HTTP. That means, your computer chooses a random port to connect from, and connects to port #80 on the server.
Unix and the shell
The majority of the Web (The Internet, even) runs on an OS called Linux (a Unix variant), instead of something like Windows. Unix systems possess a command-line interface, running a program called a "shell", which is a direct interface to the system. The shell accepts input, one command at a time. You type text in, and it spits out the out put of the command.
Secure Shell
SSH allows you to do this securely. All data traffic is encrypted using a well-studied published "public-key" cryptographic system. (In fact, it was major news when a vulnerability was discovered in a supporting encryption scheme, see these advisories).
SSH is a protocol commonly running on port 22. Anyone with a computer on the Internet (not behind a firewall) can run an SSH server, and allow users to connect to it and execute commands.
The majority of systems administrators and software developers using Unix on the server use SSH to configure, control, and upload programs to that server (located in some data center somewhere).
More
There are many many more details to all of this. Any term or acronym above can be typed into Wikipedia for pretty comprehensive information. There are plenty of books on Unix, Networking, and Web programming.
SSH is originally a secured replacement for telnet. The need for SSH arose from the fact that telnet does not support encryption and therefore everything (commands, output and password) was plainly visible on the network for all to see.
Because in the beginning SSH encryption (based on key exchange) was supposed to be strong (and it was indeed a marked improvement), and was open source, it took off rapidly and several extensions to the protocol were added, especially in the domain of remote file manageent and transfer.
In addition, SSH is used in tunelling and port forwarding configurations.
In the domain of file copy there are several options.
SCP: cp (copy). Inspired by rcp, an early file transfer extension to ssh.
SFTP: SSH File Transfer Protocol, a newer SSH extension to support File copy and browsing (but not really like FTP with 2 ports). It is more feature rich than both scp and ftp. Think of it as a remote file system protocol (however, however somewhat slower than scp).
FTPS: FTP over TLS/SSL. Needs 2 ports like ftp, one for command and one for data. Both connections can be encrypted.
Secure FTP. Real FTP tunelled over SSH.
The site to which you will need to connect probably offers SFTP. You just need to declare the remote server connection configuration in Filezilla site manager. You will need to provide the server ip address or name, the SSH server port, usually 22 but there are other possibilities (you should have been provided with this info) and select sftp as server type). When the connection is established, accept the public key and that should be it.
You can then drop your devs on the remote server.
OS choice
You shall first make a kind of choice between 2 worlds (MS or Linux).
Provided that the Linux community is somehow significantly less reluctant to share explanations. Also you will loose less time by choosing one or the other one, avoiding to wonder the same questions twice, with different answers depending on which OS you chose.
I experienced both, starting to search for solutions in the MS world, that I knew. Big mistake, loss of time. Then I changed, too late, to the Linux world. So I would advice to go straight to the linux OS for learning. Really many distributions for this. I would advice Debian (opened, user friendly, simple, safe, huge community) but you'll get as many proposals as there are admin.
OS understanding
http://www.linuxfromscratch.org/lfs/
http://www.ibm.com/developerworks/library/l-bash.html
http://tldp.org/LDP/abs/html/
Specific Questions about SSH
It depends a lot on the system you will choose but you could easily build a small client and a small server, then configure both and use ssh. Your 2 servers could even be hosted on the same machine, locally if you wish. Then you will learn how to set up the ssh-client side (often called ssh_config) and the ssh server side (often named sshd_config, with "d" standing for daemon).
Here you can find explanations about ssh for both worlds :
http://support.suso.com/supki/SSH_Tutorial_for_Linux
Some keywords for your google searches
List_of_TCP_and_UDP_port_numbers
ssh-keygen : encrypted keys (private/public),
ssh-add ssh agent
Gentoo keychain
and later but soon if you administrate your server on your own
The two main ones :
1) iptables
You may start with this and then go further with that one
2) fail2ban
this is a complement tool for which you'll find easily plenty of docs
...
Have fun :-)
EDIT: you can easily experience a Linux machine hosted in a windows OS, using virtualization (virtualbox, vm-ware..). It's a safe start and offer a good payback for this time investment. It would allow you to host as many machines (for example one linux server and one linux client) as you wish, in the limits of your HD room.
I assume you need to learn shell scripting. I recommend this book.
Filezilla is a FTP client. Try Putty - free SSH Client. And of course you need Linux server.
If you want to learn about SSH in depth then may I advise you this book SSH: The Secure Shell The Definitive Guide
See here for more info: http://www.snailbook.com/
I've read the book and learned really a lot. It teaches you all about setting up servers, clients, key agents and various (practical) applications.
Imagine you have a web site that you want to send a lot of data. Say 40 files totaling the equivalence of 2 hours of upload bandwidth. You expect to have 3 connection losses along the way (think: mobile data connection, WLAN vs. microwave). You can't be bothered to retry again and again. This should be automated. Interruptions should not cause more data loss than neccessary. Retrying a complete file is a waste of time and bandwidth.
So here is the question: Is there a software package or framework that
synchronizes a local directory (contents) to the server via HTTP,
is multi-platform (Win XP/Vista/7, MacOS X, Linux),
can be delivered as one self-contained executable,
recovers partially uploades files after interrupted network connections or client restarts,
can be generated on a server to include authentication tokens and upload target,
can be made super simple to use
or what would be a good way to build one?
Options I have found until now:
Neat packaging of rsync. This requires an rsync (server) instance on the server side that is aware of a privilege system.
A custom Flash program. As I understand, Flash 10 is able to read a local file as a bytearray (indicated here) and is obviously able to speak HTTP to the originating server. Seen in question 1344131 ("Upload image thumbnail to server, without uploading whole image").
A custom native application for each platform.
Thanks for any hints!
Related work:
HTML5 will allow multiple files to be uploaded or at least selected for upload "at once". See here for example. This is agnostic to the local files and does not feature recovery of a failed upload.
Efficient way to implement a client multiple file upload service basically asks for SWFUpload or YUIUpload (Flash-based multi-file uploaders, otherwise "stupid")
A comment in question 997253 suggests JUpload - I think using a Java applet will at least require the user to grant additional rights so it can access local files
GearsUploader seems great but requires Google Gears - that is going away soon
We need to move off traditional FTP for security purposes (it transmits it's passwords unencrypted). I am hearing SSH touted as the obvious alternative. However I have been driving FTP from an ASP.NET program interface to automate my web-site development, which is now quite a highly web-enabled process.
Can anyone recommend a secure way to transfer files around which has a program interface that I can drive from ASP.NET?
sharpssh implements sending files via scp.
the question has three subquestions:
1) choosing the secure transfer protocol
The secure version of old FTP exists - it's called FTP/SSL (plain old FTP over SSL encrypted channel). Maybe you can still use your old deployment infrastructure - just check whether it supports the FTPS or FTP/SSL.
You can check details about FTP, FTP/SSL and SFTP differences at http://www.rebex.net/secure-ftp.net/ page.
2) SFTP or FTP/SSL server for Windows
When you choose whether to use SFTP or FTPS you have to deploy the proper server. For FTP/SSL we use the Gene6 (http://www.g6ftpserver.com/) on several servers without problems. There is plenty of FTP/SSL Windows servers so use whatever you want. The situation is a bit more complicated with SFTP server for Windows - there is only a few working implementations. The Bitvise WinHTTPD looks quite promising (http://www.bitvise.com/winsshd).
3) Internet File Transfer Component for ASP.NET
Last part of the solution is secure file transfer from asp.net. There is several components on the market. I would recommend the Rebex File Transfer Pack - it supports both FTP (and FTP/SSL) and SFTP (SSH File Transfer).
Following code shows how to upload a file to the server via SFTP. The code is taken from our Rebex SFTP tutorial page.
// create client, connect and log in
Sftp client = new Sftp();
client.Connect(hostname);
client.Login(username, password);
// upload the 'test.zip' file to the current directory at the server
client.PutFile(#"c:\data\test.zip", "test.zip");
// upload the 'index.html' file to the specified directory at the server
client.PutFile(#"c:\data\index.html", "/wwwroot/index.html");
// download the 'test.zip' file from the current directory at the server
client.GetFile("test.zip", #"c:\data\test.zip");
// download the 'index.html' file from the specified directory at the server
client.GetFile("/wwwroot/index.html", #"c:\data\index.html");
// upload a text using a MemoryStream
string message = "Hello from Rebex SFTP for .NET!";
byte[] data = System.Text.Encoding.Default.GetBytes(message);
System.IO.MemoryStream ms = new System.IO.MemoryStream(data);
client.PutFile(ms, "message.txt");
Martin
We have used a variation of this solution in the past which uses the SSH Factory for .NET
The traditional secure replacement for FTP is SFTP, but if you have enough control over both endpoints, you might consider rsync instead: it is highly configurable, secure just by telling it to use ssh, and far more efficient for keeping two locations in sync.
G'day,
You might like to look at ProFPD.
Heavily customisable. Based on Apache module structure.
From their web site:
ProFTPD grew out of the desire to have a secure and configurable FTP server, and out of a significant admiration of the Apache web server.
We use our adapted version for large scale transfer of web content. Typically 300,000 updates per day.
HTH
cheers,
Rob