I want to download a file from Mainframe server to Linux machine using SFTP command and want to preserve EOL. It seems that the actual file does not have EOL character, where as SFTP process adds EOL character while downloading it to Linux server.
Is there any option to download the file in Binary mode using SFTP command?
I tried to download data from Mainframe server using FTP and SFTP. FTP with Binary mode is preserving actual content of original file without any modification, whereas FTP with ASCII mode and SFTP commands are modifying the content of the file by adding EOL or any other special characters.
I got it worked by enabling binary mode of transfer in SFTP session:
ls /+mode=binary
The default was set to ASCII.
sftp> ls /+ /+/clientcp=ISO8859-1 /+/error.log /+/loglevel=I /+/mode=text /+/servercp=IBM-1047 /+/trim
sftp> bye
I am now able to match the size of file with FTP binary and SFTP binary.
I haven't had issues with binary files and sftp. If you want to download a text file, you should use scp, unless it's in ASCII, in which case I would think sftp should work fine.
Are you trying to download a file from the hierarchical file system or from a dataset? I don't think datasets are supported with sftp (you would use FTP for that or Secure FTP).
On z/OS there are basically three ways to move content based on your question. First is FTP, second is scp and the third is sftp.
Also, its important to identify which fileset you are trying to access; MVS or USS. MVS files are different than traditional *nix file systems. MVS files are generally fixed or variable in nature. As such, they do not rely on line terminators like \n or \r to terminate the line. The record length is available at the time of reading. USS files would have line terminators.
FTP is services are provided by a daemon that is part of the Communication Server product. It provides most modes of transfer as well as extensions to deal with platform specific items as they relate to the MVS name space. You can also access Unix files as well. Conversions like ASCII (IBM-1047 -> UTF-8) are supported or binary (just move the data and don't touch it).
scp is delivered as part of the OpenSSH implementation on z/OS and it always transfers in a character mode and will assume an encoding change to / from EBCDIC to ASCII. Its an unfortunate implementation but it is what it is. This is due to the fact that the native code page on z/OS in EBCDIC in some form. Don't expect transfers with scp to move data without conversion. Only files in the USS file system are accessible. No MVS datasets.
sftp uses the secure services of OpenSSH but acts like ftp. Depending on your client you can set the mode to transfer ascii (conversion) or binary. My client on Mac OS X will not allow ascii. Probably a client limitation. sftp also only allows you to move files that reside or are destined for the USS file domain.
To answer your question about binary in sftp the answer is yes, I use it frequently to move files to and from z/OS using binary to avoid automatic conversion. Make sure you are transferring using binary transfer and not ascii when using sftp.
Related
I get a valid connection with phpseclib but because of the server's requirements I must issue a change remote directory command, $sftp->chdir($dir="//ARTDONE.G9876TT1"), to this directory, (exact format, not the actual name). This change directory command works with Putty's psftp.exe as "CD //ARTDONE.G9876TT1" in windows and with WinSCP's "go to this folder GUI input" but not with PHPSECLIB's sftp object method. Is there something about this directory format that needs to change when using phpseclib? The error message is "permission denied", but I get that same message for any other navigation commands.
Is there a way to issue literal sftp commands with phpseclib sftp?
Or can I use $ssh->exec("CD //ARTDONE.G9876TT1") in some way within the $sftp object that I cannot currently imagine?
The phpseclib appends / to the path in SFTP::chdir call. I believe this is what your server does not like.
Note that SFTP does not even use the concept of a working directory. It's faked locally by phpseclib (and other clients like WinSCP or OpenSSH). So you do not really need to use SFTP::chdir. You might instead use absolute paths in all phpseclib API calls. Alternatively, just setting SFTP::pwd has the same effect as calling SFTP::chdir, except that you will bypass the validation that causes you the troubles.
Accessing z/OS Data Sets via SFTP/FTP
Appending a / surely breaks the access. The OP is accessing an IBM z/OS system running an SFTP server.
IBM z/OS
z/OS is kind a hybrid operating system having a traditional MVS based "kernel" (not really named "kernel" in the doc), and a XPG 4.2 compliant UNIX kernel running in parallel. The UINX side supports file systems with directories and files. The MVS side has a completely different "file system", based on data sets which are named in a non-hierarchal system.
The UNIX file system on z/OS
There is not much to say about the UNIX file system on z/OS. Is it XPG compliant, thus the usage is not different to any other UNIX lik system.
The MVS Data Sets on z/OS
As said above, there is the traditional MVS Data Set based "file system" on z/OS, which is quite different to much you know about files and directories on UNIX system.
Disk Space on z/OS is assigned to MVS data sets. Data sets are named using dot separates names, that can be up to 44 characters long. The parts between two dots can be up to 8 characters long.
Examples:
ARTDONE.G9876TT1
ARTDONE.NEXT.DATA.SET
ARTDONE.NEXT.ANOTHER.ONE
SYS1.LINKLIB
ZUSER.SOURCE.REXX
What seems to be a hierarchy in the first three examples, is not. They are unrelated from the physical point of view, though related in a logical.
Note: Slashed / are not valid in MVS data set names.
SFTP/FTP servers on z/OS
SFTP/FPT servers in z/OS mimic the client side view of directories and files when accessing MVS Data Sets in that the dots in the names are kind of treated like slashes in UNIX. I.e. they support pwd and cd based on the dots.
Example:
cd //ARTDONE.NEXT sets the current working directory to ARTDONE.NEXT. A ls the lists all data sets, of which the name starts with ARTDONE.NEXT, i.e.
ARTDONE.NEXT.DATA.SET
ARTDONE.NEXT.ANOTHER.ONE
but not ARTDONE.G9876TT1.
But how would the server know whether a client side "directory" access is meant to access the UNIX or the MVS data world? The // at the beginning of the parameter passed to the server indicates the server shall switch to the MVS data set world.
Conclusion
client side software should be careful when verifying paths that might be sent to z/OS servers. Accept // as a special indicator. Do not append / in all cases.
More Detailes
There is much more behind this topic than can be described here. Read IBM documentation on z/OS if interested. I would recommend Introduction to the New Mainframe: z/OS Basics as a starter.
I am using a z/OS java jar which requires a keystore and truststore file.
I have FTP'd all of the files to the z/OS Unix directory(Mainframe).
When executing the jar, I receive an "invalid keystore file format" error. When I use the keytool on my local machine on the keystore file, the command is successful. When using the keytool command on the remote (zOS Unix), I am getting the format error. The files are being transmitted in binary format.
Is there a particular way that I should be transmitting the file to MF Unix?
There are three primary way I use to transfer files to z/OS USS depending on whether I want conversion or not (from utf-8 to ibm-1047 for instance).
With conversion:
scp works and will translate the files characters while transferring. Note: this is an IBM behaviour that you do not normally see when transferring from distributed system to distributed system. It's really annoying.
ftp with ascii mode
Without Conversion:
ftp with transfer mode set to binary
sftp
Choose the option that makes the most sense based on the client in your environment.
I have a requirement to perform checksum (for data integrity) for SFTP. I was hoping this could be done during the SFTP file transfer - I realize this could be product dependent (FYI: using CLEO VLTrader), but was wondering if this is customary?
I am also looking for alternative data integrity checking options that are as good (or better) than using a checksum algorithm. Thanks!
With the SFTP, running over an encrypted SSH session, there's negligible chance the file contents could get corrupted while transferring. The SSH itself does data integrity verification.
So unless the contents gets corrupted, when reading the local file or writing the remote file, you can be pretty sure that the file was uploaded correctly, if no error is reported. That implies that a risk of data corruption as about the same as if you were copying the files between two local drives.
If you would not consider it necessary to verify data integrity after copying the files from one local drive to another, then I do not think, you need to verify integrity after an SFTP transfer, and vice versa.
If you want to test explicitly anyway:
While there's the check-file extension to the SFTP protocol to calculate a remote file checksum, it's not widely supported. Particularly it's not supported by the most widespread SFTP server implementation, the OpenSSH. See What SFTP server implementations support check-file extension.
Not many clients/client libraries support it either. You didn't specify, what client/library you are using, so I cannot provide more details.
For details about some implementations, see:
Python Paramiko: How to check if Paramiko successfully uploaded a file to an SFTP server?
.NET WinSCP: Verify checksum of a remote file against a local file over SFTP/FTP protocol
What SFTP server implementations support check-file extension
Other than that, your only option is to download the file back (if uploading) and compare locally.
If you have a shell access to the server, you can of course try to run some shell checksum command (e.g. sha256sum) over a separate shell/SSH connection (or the "exec" channel) and parse the results. But that's not an SFTP solution anymore.
Examples:
Calculate hash of file with Renci SSH.NET in VB.NET
Comparing MD5 of downloaded files against files on an SFTP server in Python
What does the ASCII mode in SFTP transfer means. Does it going to transfer the file in text format.
Quoting SFTP specification regarding text (aka ASCII) access to remote files:
5.3. Determining Server Newline Convention
In order to correctly process text files in a cross platform
compatible way, newline sequences must be converted between client
and server conventions.
The SSH_FXF_TEXT_MODE file open flag (Section 8.1.1) makes it
possible to request that the server translate a file to a 'canonical'
wire format. This format uses CRLF as the line separator.
Servers for systems using other conventions MUST translate to and
from the canonical form.
Note that the "text mode" is supported since SFTP version 4 only. Many existing SFTP implementations (including the most widespread one, the OpenSSH) support SFTP version 3 only. So they do not support text/ASCII mode at all.
We have a IBM Host System Z sitting in our cellar. Now the issue is that i have no clue about Mainframes!!! (It's not USS btw.)
The Problem: How can i transfer a file from the host system to a windows machine.
Usually on UNIX systems i would just install and ssh daemon and connect to it via. a program called winscp. After that transfer the file in binary so that it does not convert something (Ultraedit and other Editors can handle this).
With the host system it seems to be a bit difficult as the original format from IBM is EBCDIC and i have no idea if there is a state of the art SFTP server program for the host. Could anybody be so kind and enlighten me? From my current expirience with IT there must be a state of the art sftp connection to that system? I appreciate any help/hints/solutions.
Thank you,
O.S
If the mainframe "sitting in [your] cellar" is running z/OS then it has Unix System Services installed. You can't have z/OS without it.
There is an SFTP package available (for free) for z/OS.
You can test to see about Unix System Services by firing up a 3270 emulator going to ISPF option 3.17, putting a forward slash (/) in the Pathname field and pressing the mainframe Enter key. Another way would be to key OMVS at a TSO READY prompt, which will start up a 3270-based Unix shell.
It is possible that USS is simply not available to you; if you're running any supported release of z/OS then USS is present. There could be concerns about supporting something outside a particular group,
Or, depending on what OS you have running on your System z, it's possible you don't have z/OS. You could have z/VM, you could have zLinux, you could have TPF. However, if you're running zLinux, you have linux, which has sftp installed, and which uses ASCII, not EBCDIC.
As cschneid says, however, if you have z/OS, you have USS. TCP/IP, among other things, won't run without it. Also note that z/OS TCP/IP has an FTP server, so you can connect that way if the FTP server is set up. If security is an issue, FTPS is supported, although it's painful to set up. With the native FTP server, you can convert from EBCDIC to ASCII when you're doing the transfer. There's also an NFS server available. And SMB as well, I believe.
And there's an FTP client available as well, so you could FTP from z/OS to your system, if you wanted to.
Maybe a better thing to do would explain what you're trying to do with the data, and what the data is, in general. You can edit files directly on the mainframe, using either TSO, ISPF, or OMVS editors. There are a lot of data types that the mainframe supports that you're not going to be able to handle on a non-z system unless you go through an export process. I'm not really clear on whether you want to convert the file to ASCII when you transfer it or not.
While the others are correct that all recent releases of z/OS have USS built-in, there's quite a bit of setup work that needs to be done in order for individual users to have access to USS capabilities like SFTP. Out of the box, you get USS "minimal mode" that just has enough of USS to support the TCP/IP stack and so forth. USS "full function mode" requires setup:
HFS filesystems need to be allocated
Your security package needs to be manage UIDs/GIDs for your users
etc etc etc
Still, with these details and with nothing more than the software you're entitled to as part of your z/OS license, you can certainly run SFTP and all the other UNIX style network services you're used to.
A good place to start is the UNIX Services Planning guide: http://publibz.boulder.ibm.com/epubs/pdf/bpxzb2c0.pdf