I'm using WinSCP .NET assembly. When I call the Session.PutFiles method, it sends the following series of commands:
TYPE A
PASV
MLSD
TYPE A
PASV
STOR myfile
Is there a way to tell it NOT to send the MLSD? (MLSD requests the contents of the remote directory be sent back.) At the very least, I don't need this information so it's just wasting bandwidth. I don't even know how I would access it -- maybe WinSCP is doing something with it internally? What worries me more, though, is that I was given very specific specs about the series of FTP commands that I was supposed to send, which includes several non-standard commands, apparently the site at the other end has a customized FTP server. So I don't want an extra command to screw things up.
In the latest version, with default transfer settings, WinSCP does not use the MLSD command.
It's used only with OverwriteMode.Resume or OverwriteMode.Append to retrieve attributes of the remote file.
Also, WinSCP issues the MLSD command once for every destination directory (not for each file).
Related
I get a valid connection with phpseclib but because of the server's requirements I must issue a change remote directory command, $sftp->chdir($dir="//ARTDONE.G9876TT1"), to this directory, (exact format, not the actual name). This change directory command works with Putty's psftp.exe as "CD //ARTDONE.G9876TT1" in windows and with WinSCP's "go to this folder GUI input" but not with PHPSECLIB's sftp object method. Is there something about this directory format that needs to change when using phpseclib? The error message is "permission denied", but I get that same message for any other navigation commands.
Is there a way to issue literal sftp commands with phpseclib sftp?
Or can I use $ssh->exec("CD //ARTDONE.G9876TT1") in some way within the $sftp object that I cannot currently imagine?
The phpseclib appends / to the path in SFTP::chdir call. I believe this is what your server does not like.
Note that SFTP does not even use the concept of a working directory. It's faked locally by phpseclib (and other clients like WinSCP or OpenSSH). So you do not really need to use SFTP::chdir. You might instead use absolute paths in all phpseclib API calls. Alternatively, just setting SFTP::pwd has the same effect as calling SFTP::chdir, except that you will bypass the validation that causes you the troubles.
Accessing z/OS Data Sets via SFTP/FTP
Appending a / surely breaks the access. The OP is accessing an IBM z/OS system running an SFTP server.
IBM z/OS
z/OS is kind a hybrid operating system having a traditional MVS based "kernel" (not really named "kernel" in the doc), and a XPG 4.2 compliant UNIX kernel running in parallel. The UINX side supports file systems with directories and files. The MVS side has a completely different "file system", based on data sets which are named in a non-hierarchal system.
The UNIX file system on z/OS
There is not much to say about the UNIX file system on z/OS. Is it XPG compliant, thus the usage is not different to any other UNIX lik system.
The MVS Data Sets on z/OS
As said above, there is the traditional MVS Data Set based "file system" on z/OS, which is quite different to much you know about files and directories on UNIX system.
Disk Space on z/OS is assigned to MVS data sets. Data sets are named using dot separates names, that can be up to 44 characters long. The parts between two dots can be up to 8 characters long.
Examples:
ARTDONE.G9876TT1
ARTDONE.NEXT.DATA.SET
ARTDONE.NEXT.ANOTHER.ONE
SYS1.LINKLIB
ZUSER.SOURCE.REXX
What seems to be a hierarchy in the first three examples, is not. They are unrelated from the physical point of view, though related in a logical.
Note: Slashed / are not valid in MVS data set names.
SFTP/FTP servers on z/OS
SFTP/FPT servers in z/OS mimic the client side view of directories and files when accessing MVS Data Sets in that the dots in the names are kind of treated like slashes in UNIX. I.e. they support pwd and cd based on the dots.
Example:
cd //ARTDONE.NEXT sets the current working directory to ARTDONE.NEXT. A ls the lists all data sets, of which the name starts with ARTDONE.NEXT, i.e.
ARTDONE.NEXT.DATA.SET
ARTDONE.NEXT.ANOTHER.ONE
but not ARTDONE.G9876TT1.
But how would the server know whether a client side "directory" access is meant to access the UNIX or the MVS data world? The // at the beginning of the parameter passed to the server indicates the server shall switch to the MVS data set world.
Conclusion
client side software should be careful when verifying paths that might be sent to z/OS servers. Accept // as a special indicator. Do not append / in all cases.
More Detailes
There is much more behind this topic than can be described here. Read IBM documentation on z/OS if interested. I would recommend Introduction to the New Mainframe: z/OS Basics as a starter.
I am looking for JCL Script/Procedures in mainframe which can facilitate file transfer from Unix server to Mainframe.I am required to do FTPS for the Outbound Jobs (pull the file from UNIX server to mainframe Host).
Rather than a JCL, just do it a shell script. Here is a good site on using such commands:
https://blog.eduonix.com/shell-scripting/how-to-automate-ftp-transfers-in-linux-shell-scripting/
Once you have that working in the shell script in USS, you should be able to call the shell script from a JCL so you can execute it on a scheduled batch job if you need it.
Kenny's suggestion is fairly reasonable. IBM's documentation on how to write JCL for FTP(S)-related tasks is available in their "z/OS Communications Server: IP User's Guide and Commands" publication, IBM Publication No. SC27-3662. The current revision appears to be SC27-3662-30, but later revisions are possible. You can easily find this publication online, and make sure you don't skip the section beginning with the title "Submitting FTP requests in batch." Make sure you set the security options correctly (of course).
Please note that you're asking about FTPS, i.e. TLS encryption applied to either or both (preferably both) of the FTP channels (control and data). SFTP is another file transfer protocol based on SSH that z/OS also supports.
Another possible approach that you'll fairly often find available on z/OS installations is to use IBM MQ Advanced for z/OS's Managed File Transfer (MFT) feature to retrieve the file(s) using FTPS. As the name suggests, this'll be managed and have at least some error handling capabilities.
Yet another possible approach if you prefer HTTPS protocol is to use the z/OS Client Web Enablement Toolkit's HTTPS protocol enabler to fetch the file. That's a built-in, standard feature in all currently supported z/OS releases, and you can use it from a relatively simple REXX script for example. Details are available here (z/OS 2.3 variant of the documentation):
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.ieac100/ieac1-cwe-http.htm
I have a requirement to perform checksum (for data integrity) for SFTP. I was hoping this could be done during the SFTP file transfer - I realize this could be product dependent (FYI: using CLEO VLTrader), but was wondering if this is customary?
I am also looking for alternative data integrity checking options that are as good (or better) than using a checksum algorithm. Thanks!
With the SFTP, running over an encrypted SSH session, there's negligible chance the file contents could get corrupted while transferring. The SSH itself does data integrity verification.
So unless the contents gets corrupted, when reading the local file or writing the remote file, you can be pretty sure that the file was uploaded correctly, if no error is reported. That implies that a risk of data corruption as about the same as if you were copying the files between two local drives.
If you would not consider it necessary to verify data integrity after copying the files from one local drive to another, then I do not think, you need to verify integrity after an SFTP transfer, and vice versa.
If you want to test explicitly anyway:
While there's the check-file extension to the SFTP protocol to calculate a remote file checksum, it's not widely supported. Particularly it's not supported by the most widespread SFTP server implementation, the OpenSSH. See What SFTP server implementations support check-file extension.
Not many clients/client libraries support it either. You didn't specify, what client/library you are using, so I cannot provide more details.
For details about some implementations, see:
Python Paramiko: How to check if Paramiko successfully uploaded a file to an SFTP server?
.NET WinSCP: Verify checksum of a remote file against a local file over SFTP/FTP protocol
What SFTP server implementations support check-file extension
Other than that, your only option is to download the file back (if uploading) and compare locally.
If you have a shell access to the server, you can of course try to run some shell checksum command (e.g. sha256sum) over a separate shell/SSH connection (or the "exec" channel) and parse the results. But that's not an SFTP solution anymore.
Examples:
Calculate hash of file with Renci SSH.NET in VB.NET
Comparing MD5 of downloaded files against files on an SFTP server in Python
I was looking for list of available options for ftp quote site which will allow to do lot of stuff like executing commands on the target system. Like the below one can be used to submit jobs on mainframes.
quote site filetype=jes
put filetoexecute.jcl
I know that there are similar options for unix environment as well. Is there any list of available options for this quote command for unix, mainframes (windows also if available) environments.
Also below is an extra question based on these.
Is there any way to execute ca7 commands from this ftp? If not I was looking into one rexx example which will execute the ca7 commands passed as arguments. But this is failing with "CA-7 RECEIVER NOT FOUND" error.
PARSE UPPER ARG COMMAND
ADDRESS CA7 COMMAND
SAY 'RC=' RC
X=QUEUED()
SAY 'QUEUED() =' X
DO I=1 TO X
PULL LINE
LINE2=SUBSTR(LINE,2)
SAY LINE2
END
Also the below command is failing with the same failure.
ADDRESS CA7 "'LQ,SEQ=JOB,JOB=*'"
I have checked ca7 manual and don't know how to make sure that CA7 environment is configured to execute the above commands.
Can you please help?
The IBM FTP server supports HELP SITE; this gives you all of the operands that the SITE command supports. You can issue STAT to get the current SITE values.
This is specific to the IBM FTP server. Each FTP server is different, and they may or may not implement the SITE command. I suggest looking at the doc for the FTP server to find if they support SITE and STAT, or similar.
Every month we send reports to a server using FTP. We run a query on a database to create the files then use the ftp functionality in LabVIEW to do the transfer. This runs on a Windows system.
This works fine but now we have to switch to using SFTP and the CopSSH package has been recommended. As LabVIEW has no native SFTP functionality we are looking at how we can use the sftp.exe application from CopSSH.
From the command prompt we have set up the encryption and made the initial connection using sftp username#host and entered the password. This has been confirmed by the team on the server side so connection to the server is set up. Now we just use sftp username#host and no password is required.
Where we are struggling is how to initiate the transfer from our LabVIEW code. We are able to call system commands using the System Exec VI but is there a way to pass a list of functions to the SFTP executable?
The commands used to transfer the files when we type it at the command prompt are:
sftp username#host
put c:/Data/File1.txt remoteFile1
put c:/Data/File2.txt remoteFile2
put c:/Data/File3.txt remoteFile3
quit
This works from the command prompt but I am looking to just call the sftp executable with a list of files to transfer. I don't think this would be specific to LabVIEW as you could use a batch file to run from a scheduled job.
LabVIEW can call ActiveX and .net but we really need to use this specific application.
I have been using WinSCP which has a command line version, winscp.com. It supports sftp and allows synchronize, keepuptodate, get, put and delete on folders and files. One word of warning, keepuptodate depends on an unbroken connection. Although WinSCP can remake a connection automatically, keepuptodate cannot. I suspect it is based on Microsoft's .NET SystemIO FileSystemWatcher. I therefore do a regular synchronize to keep a mirror of my source folder tree on the remote target.
If copssh's sftp.exe is a command line utility, and System Exec in your version of LabVIEW has the 'standard input' terminal (present at least since 8.5), you should be able to simply wire the commands you want sftp.exe to run into the standard input terminal.
If that doesn't work for some reason, could you use PuTTY instead of copssh? The documentation for PuTTY's PSFTP component says that it can execute a sequence of commands in a script file using the -b command line switch, e.g.
psftp user#hostname -b myscript.scr
so you could have your LabVIEW program create the script file then run it with System Exec.
You are mixing SSH and SFTP. SSH opens a secure connection, but SFTP is a separate protocol which is run over SSH connection and requires a separate tunnel. In OpenSSH (and it's Windows Port, copSSH) it's sftp.exe application that does SFTP.
Now about FTP vs SFTP. Please check an article that explains the difference between SFTP and FTP(S). If LabView supports FTP, this doesn't help you when you need to perform SFTP transfers.
I don't know whether you can use external ActiveX controls in LabView. If you can, you are welcome to check our SFTP ActiveX control, that will let you do the transfer. If all you can do is call external application, then you'd have to use copSSH's sftp.exe.